Skip to content

[AMORO-3781] Save the last completion time for each cleanup operation performed on each optimization table. - #3802

Merged
zhoujinsong merged 3 commits into
apache:masterfrom
zhangwl9:AMORO-Save-lastTime-dev
Nov 20, 2025
Merged

[AMORO-3781] Save the last completion time for each cleanup operation performed on each optimization table.#3802
zhoujinsong merged 3 commits into
apache:masterfrom
zhangwl9:AMORO-Save-lastTime-dev

Conversation

@zhangwl9

@zhangwl9 zhangwl9 commented Sep 26, 2025

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Close #3781.

Brief change log

To optimize table cleanup operations (Dangling-Delete_file cleanup, data expiration, orphan_file cleanup,snapshotsFile cleanup), the last cleanup time is saved. This ensures that after an AMS service restart, if a cleanup operation was completed just before the restart, the next cleanup operation will only be performed after exceeding the respective execution interval.
The execution flow is as follows:

  • During the initialization of the ams service, retrieve the last execution time for each cleanup operation of every table to be optimized from the table_runtime_state table.

  • Check whether the time interval since the last cleanup exceeds the cleanup operation execution interval.

  • For optimization tables exceeding the execution interval, execute the cleanup operation.

  • Otherwise, schedule for the next round of execution.

  • Since each table's cleanup operation takes varying durations, after a table's cleanup completes, update the cleanup completion time in the table_runtime_state table.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@github-actions github-actions Bot added the module:ams-server Ams server module label Sep 26, 2025
@zhangwl9
zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch from bce7e0d to 8162476 Compare September 28, 2025 03:28
@codecov-commenter

codecov-commenter commented Sep 28, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.78431% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.93%. Comparing base (fd44c44) to head (2aba806).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...amoro/server/scheduler/PeriodicTableScheduler.java 40.81% 28 Missing and 1 partial ⚠️
...apache/amoro/server/table/DefaultTableRuntime.java 88.46% 1 Missing and 2 partials ⚠️
...er/inline/DanglingDeleteFilesCleaningExecutor.java 0.00% 2 Missing ⚠️
.../server/scheduler/inline/DataExpiringExecutor.java 0.00% 2 Missing ⚠️
.../scheduler/inline/OrphanFilesCleaningExecutor.java 0.00% 2 Missing ⚠️
...er/scheduler/inline/SnapshotsExpiringExecutor.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3802      +/-   ##
============================================
- Coverage     29.45%   28.93%   -0.53%     
+ Complexity     4236     3870     -366     
============================================
  Files           670      631      -39     
  Lines         55245    50743    -4502     
  Branches       6983     6480     -503     
============================================
- Hits          16273    14682    -1591     
+ Misses        37778    35011    -2767     
+ Partials       1194     1050     -144     
Flag Coverage Δ
core 28.93% <60.78%> (+0.11%) ⬆️
trino ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@zhangwl9
zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch 4 times, most recently from 789c5bd to 7b1006d Compare November 5, 2025 02:59
@zhangwl9
zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch from 7b1006d to e69848b Compare November 11, 2025 10:46
@zhangwl9

zhangwl9 commented Nov 11, 2025

Copy link
Copy Markdown
Contributor Author

I have completed the logic modification to save cleanup times to the table_runtime_state table. Could you please review the code at your convenience? Thank you very much. @czy006 @baiyangtx

@zhangwl9
zhangwl9 force-pushed the AMORO-Save-lastTime-dev branch from e69848b to 7b21fe0 Compare November 13, 2025 05:55
张文领 added 3 commits November 19, 2025 20:14
… each optimization table.

# Conflicts:
#	amoro-ams/src/main/resources/mysql/upgrade.sql
@klion26
klion26 force-pushed the AMORO-Save-lastTime-dev branch from a59efa7 to af05e8e Compare November 19, 2025 12:14

@zhoujinsong zhoujinsong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@zhoujinsong

Copy link
Copy Markdown
Contributor

Thanks a lot for the contribution!

@zhoujinsong
zhoujinsong merged commit 24705a5 into apache:master Nov 20, 2025
8 checks passed
@zhangwl9
zhangwl9 deleted the AMORO-Save-lastTime-dev branch November 27, 2025 02:04
@zhangwl9 zhangwl9 changed the title Save the last completion time for each cleanup operation performed on each optimization table. [AMORO-3781] Save the last completion time for each cleanup operation performed on each optimization table. Dec 2, 2025
wardlican pushed a commit to wardlican/amoro that referenced this pull request Jan 15, 2026
… each optimization table. (apache#3802)

* Save the last completion time for each cleanup operation performed on each optimization table.

# Conflicts:
#	amoro-ams/src/main/resources/mysql/upgrade.sql

* Store the execution time of each cleanup operation for the optimization table in the table_runtime_state table.

* fixup style

---------

Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn>
czy006 added a commit that referenced this pull request Jan 21, 2026
…lave mode is enabled. (#3846)

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [AMORO-3850] Fix openapi sdk build and refine the package name (#3847)

* rename the sdk pkg

* version and artifactId

* [HotFix] Change the uri configuration name in terminal  (#3844)

Change the uri configuration in terminal

* [AMORO-3851] Separate spark-3.3 and spark-3.5 modules (#3848)

* separate

* java17

* build

* GA

* jdk17 tests

* fix conflicts

* <java.source.version>17</java.source.version>

* JDK8 + Spark-3.5

* revert JDK17

* [AMORO-3857] Make the metadata file name conform to the Iceberg spec (#3858)

Each version of table metadata is stored in a metadata folder under the table’s base location using a naming scheme that includes a version and UUID: <V>-<random-uuid>.metadata.json.

* [AMORO-3850][FOLLOWUP] Add GA for OpenAPI SDK (#3870)

[AMORO-3850][FOLLOWUP] Add GA for openAPI SDK

* [AMORO-3864] Bump spark 3.5 version to 3.5.7 (#3860)

Bump spark 3.5 version to 3.5.7

* [AMORO-3852][BUILD] Enhance build system (#3849)

* Build project with fixed version

check-latest: false

GA

* docs

* shell

* [AMORO-3851][FOLLOWUP] Using scala.binary.version for Spark dependency artifactId (#3861)

Using scala.binary.version for dependency artifactId

* [AMORO-3851][FOLLOWUP] Fix and simplify spark versions management (#3874)

* Fix iceberg-spark artifactId in amoro-mixed-spark-3-common

* fix ut conflicts

* [AMORO-3863] Add script to reformat code (#3862)

* format

* spark profiles

* [AMORO-3852] Using Maven Wrapper (#3879)

* save

* 3.9.9

* mvnw

* slash

* maven 3.9.11

* maven wrapper 3.3.4

* [AMORO-3880] Make amoro-openapi-sdk standalone (#3881)

* [AMORO-3880] Make amoro-openapi-sdk standalone

* save

* [AMORO-3891] Bump netty version to 4.1.128.Final to fix CVE (#3892)

Bump netty version to 4.1.128.Final to fix CVE

* [AMORO-3890]Bump flink versions from 1.16.2/1.17.1/1.20.0 to 1.16.3/1.17.2/1.20.3 (#3889)

bump

* [AMORO-3880][FOLLOWUP] Fix pull request paths for OpenAPI SDK CI (#3903)

* Bump orc-core version from 1.8.3 to 1.9.7 (#3900)

* [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 (#3884)

* Bump Paimon 1.1.1 to 1.2.0

* [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0

---------

Co-authored-by: Xu Bai <xuba@apache.org>

* [AMORO-3863][FOLLOWUP] Support to reformat pom (#3882)

Support to reformat pom

format trino

* [AMORO-3872] Support to customize basic authentication implementation (#3871)

* support to customize basic auth"

* save

* docs

* common

* address comments

* nit

---------

Co-authored-by: Xu Bai <xuba@apache.org>

* [AMORO-3907] Bump kubernetes-client version to 6.13.5 (#3908)

Bump kubernetes-client version to 6.13.5

* [AMORO-3875] Support to build on Spark Scala-2.13 (#3878)

* scala binary version

save

bin

profile

GA

revert paimon ams

api compatiblity

fix flink scala

style

save

save

rewrite by scala

save

conflicts

dependency

save

save

scala paimon

ignore paimon

idea

* nit

* nit

* save

* revert ci change

* [AMORO-3875][FOLLOWUP] Fix optimizer-spark docker image build due to artifactId change (#3912)

* fix

save

* test

* Revert "test"

This reverts commit 6c38dbd.

* [AMORO-3863][FOLLOWUP] Enable format-mixed-format-trino profile on JDK17+ automatically  (#3906)

reformat

* [AMORO-3848][FOLLOWUP] Prompt spark profile in docker build (#3904)

Prompt spark profile in docker build

* [AMORO-3918][INFRA] Add dependencies check CI (#3917)

* [AMORO-3931] Exclude curator and zookeeper deps (#3932)

* save

deps

exclude jline

* nit

* [AMORO-3945][DOCS] Add docs for REST API (#3944)

docs

combine

* [AMORO-3873] Support Bearer/JWT authentication (#3905)

* save

* save

* save

* save

* Save

* asve

* rename package

* comments

* basic

* remove token from log

* [AMORO-3938] change file_content_b64 column type to CLOB type (#3939)

* [AMORO-3933] Fix Playground demo failure due `PartitionExpressionForMetastore class not found` (#3935)

* Fix Playground demo failure due PartitionExpressionForMetastore class not found

* Revert "Fix Playground demo failure due PartitionExpressionForMetastore class not found"

This reverts commit fea6ddc.

* add for runtime hive-exec

---------

Co-authored-by: Xu Bai <xuba@apache.org>

* Save the last completion time for each cleanup operation performed on each optimization table. (#3802)

* Save the last completion time for each cleanup operation performed on each optimization table.

# Conflicts:
#	amoro-ams/src/main/resources/mysql/upgrade.sql

* Store the execution time of each cleanup operation for the optimization table in the table_runtime_state table.

* fixup style

---------

Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn>

* nit: remove duplicate code (#3957)

NIT: Remove duplicate code

* [AMORO-3873][FOLLOWUP] Rename rest auth bearer type to JWT (#3953)

* [AMORO-3968] Update the thrift api compile command to use amoro shaded thrift (#3967)

* relocate generated thrift code to use amoro shaded thrift

* docs

* save

* profile

* [AMORO-3961] Filter null key and value for Configurations::toMap (#3962)

filter null key value

* nit: remove unused code (#3959)

remove unused code

* [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 (#3975)

* [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5

This PR upgrades the default Spark version from 3.3 to 3.5 in the Amoro project.

Changes:
- pom.xml: Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5
- pom.xml (hadoop2 profile): Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5
- docker/build.sh: Update SPARK_VERSION from 3.3.3 to 3.5.7
- docker/optimizer-spark/Dockerfile: Update ARG SPARK_VERSION from 3.3.3 to 3.5.7
- .github/workflows/docker-images.yml: Update Spark optimizer matrix from 3.3.3 to 3.5.7
- README.md: Update Spark optimizer default version documentation

Closes #3972

* Keep Spark 3.3 as default for hadoop2 profile

Address review feedback from @turboFei:
For hadoop2 profile, keep using the legacy Spark 3.3 by default
for better compatibility with Hadoop 2.x.

* Fix Spark version parameter name in README.md

Address review feedback from @turboFei:
Change -Dspark-optimizer.spark-version to -Dspark.version
This was missed in PR #3874.

* [AMORO-3966][Helm] Support custom volumes and volumeMounts (#3965)

[Improvement][Helm] Support custom volumes and volumeMounts

* [AMORO-3970] Update README.md with correct Spark versions for Mixed format (#3978)

* [AMORO-3977] Combine amoro-site to prevent docs loose sync (#3979)

* clone site

* remove invalid soft links

* docs soft link

* Update site/README.md with clear structure documentation

- Reorganized documentation to show the versioned and non-versioned content structure
- Provided a clear directory tree representation of the site structure
- Updated instructions for running the documentation site locally
- Added section on testing both sites together

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update site/README.md with clear structure documentation

- Fixed references to make it clear this is part of the main repository
- Updated section titles to match the new organization
- Clarified paths for versioned and non-versioned content
- Simplified local development instructions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* github action

* ignore

* rat exclude site

* test

* test please

* Revert "test please"

This reverts commit 7c536fe.

* label

---------

Co-authored-by: Claude <noreply@anthropic.com>

* [Improvement]: Disable verification for helm-unittest plugin installation (#3993)

Disable verification for helm-unittest plugin installation

* [AMORO_3990] Add package chart instructions to README (#3991)

* [Feature] Introduce a new framework that supports scheduling for Formats and Processes (#3924)

* process service poc

* Refactor table process framework

* Optimize and refactor the process service framework & support framework test case

---------

Co-authored-by: dailiang <dailiang@bytedance.com>
Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com>
Co-authored-by: baiyangtx <xiangnebula@163.com>

* [AMORO-3981] Automatic generate amoro configuration docs  (#3982)

* auto generate ams config

* add auto generate notice

* junit 5

* convertToString

* Update CONTRIBUTING.md

---------

Co-authored-by: big face cat <731030576@qq.com>

* [AMORO-3981][FOLLOWUP] Refine the duration to string format  (#3987)

* [AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string

* Revert "[AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string"

This reverts commit 877bcb9.

* format duration with TimeUtils.formatWithHighestUnit

* largest unit

* [AMORO-1258] Support Zookeeper kerberos authentication (#3980)

* [AMORO-1258] Support Zookeeper kerberos authentication

* remove minkdc

* Revert "remove minkdc"

This reverts commit 466d632.

* docs

* [AMORO-3934] Manage com.fasterxml.jackson dependencies in dependencyManagement  (#3936)

* jackson version

* deps

* fasterxml.jackson.version

---------

Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

* [AMORO-3866] Using shaded hadoop-client-api and hadoop-client-runtime for hadoop3 (#3983)

hadoop client api and hadoop client runtime

remove

guava test

replace

exclude

remove runtime

* [AMORO-3632] Refine data expiration literal calculation for date type (#3964)

* [Feature]: data-retention, add support for partition column type Date #3632

* [AMORO-3632]: data-retention, add support for partition column type Date

* [AMORO-3632]: data-retention, add support for partition column type Date

* [AMORO-3632]: data-retention, add support for partition column type Date

* [AMORO-3632] Fix #3665: Prevent long overflow in Date expiration calculation and fix related tests

---------

* [AMORO-3885] fix missing oss.endpoint for internal mixed_iceberg catalog (#3886)

* fix missing oss.endpoint for internal mixed_iceberg catalog

* fix missing oss.endpoint for internal mixed_iceberg catalog

---------

Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
Co-authored-by: ConradJam <jam.gzczy@gmail.com>

* [AMORO-2635] Enhance table partition files list performance (#4003)

* [AMORO-2635]

* UT

* UT for fallback

* address comments

* TODO

* [AMORO-3628] Add user logo wall on the home page. (#4010)

* [AMORO-3628] Add users logo wall in the home page

* update

Co-authored-by: Claude

* [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration (#4005)

* [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration

* return

* test

* configuration

* create runtime context directly

* remove unneeded tests

* [AMORO-4011] Fix JUnit 4 tests skipped (#4017)

* add back

* test

* revert test

* deps

* [AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor (#3776)

[AMORO-3775] Add support for metadata-based refresh event in TableRuntimeRefreshExecutor

* [AMORO-3998]support DATABASE HA SERVICE (#3997)

support DATABASE HA SERVICE

Co-authored-by: dailiang <dailiang@bytedance.com>

* [Hotfix] Fix the deploy site GitHub workflow (#4020)

* Fix deploy site GitHub workflow

* Add publish information in the .asf.yaml

* [Hotfix] Enable manual triggering for site deployment workflow (#4021)

* [Hotfix]Refactor the build command in the `Publish Docker Image` GitHub workflow to reduce image size (#4023)

Refactor the build command in the Publish DOcker Image GitHub workflow to reduce image size

* [Hotfix] Fix typos (#4026)

* [AMORO-3531] Drop support for java8 (#3899)

drop support jdk8

Co-authored-by: Xu Bai <xuba@apache.org>
Co-authored-by: ConradJam <jam.gzczy@gmail.com>
Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>

* [AMORO-3940] Flink config load will cause ClassCastException when dir… (#3941)

[AMORO-3940] Flink config load will cause ClassCastException when directly return result of Yaml load

Co-authored-by: ConradJam <jam.gzczy@gmail.com>

* [AMORO-3943] Shade all third party classes for spark runtime (#3942)

* relocate

* filter

* [AMORO-4027] Bump shade plugin to fix compilation failure on JDK 17 (#4028)

(cherry picked from commit 83e5272)

* [AMORO-3973] Support spark3.4 for mixed format (#4013)

* Copy the code form spark3.5

* Optimize code

* Maintain consistency in the references to scala-compiler and scala-library across Spark 3.5, Spark 3.4, and Spark 3.3

* fixup

---------

Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn>
Co-authored-by: Xu Bai <tocreationbai@gmail.com>

* [Hotfix] Try to fix site-deploy permission problem (#4025)

* [AMORO-4022] [Improvement]: AMS Iceberg maintainer moved to the amoro-iceberg module (#4024)

[AMORO-4022] AMS Iceberg maintainer moved to the amoro-iceberg module (#4022)

* [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996)

[amoro-3994] Support Exposing AMS High Availability (HA) Status

Co-authored-by: davedwwang <davedwwang@tencent.com>

* [Feature]: Support LDAP Authentication for Dashboard Login (#4009)

Co-authored-by: davedwwang <davedwwang@tencent.com>

* [HotFix] Remove duplicate references to scala-library and scala-compiler (#4033)

Remove duplicate references to scala-library and scala-compiler

Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn>

* [HotFix] Shade third party classes for spark-3.4 runtime (#4032)

Shade third party classes for spark runtime 3.4

Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn>

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [AMORO-3998]support DATABASE HA SERVICE (#3997)

support DATABASE HA SERVICE

Co-authored-by: dailiang <dailiang@bytedance.com>

* [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996)

[amoro-3994] Support Exposing AMS High Availability (HA) Status

Co-authored-by: davedwwang <davedwwang@tencent.com>

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845

* [Subtask]: Optimize the description information for use-master-slave-mode.

---------

Co-authored-by: wardli <wardli@tencent.com>
Co-authored-by: Fei Wang <fwang12@ebay.com>
Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com>
Co-authored-by: yeatsliao <liaoyt66066@gmail.com>
Co-authored-by: Xu Bai <xuba@apache.org>
Co-authored-by: xuzifu666 <xuzifu666@gmail.com>
Co-authored-by: simonsssu <barley0806@gmail.com>
Co-authored-by: zhangwl9 <1298877813@qq.com>
Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn>
Co-authored-by: zhan7236 <76658920+zhan7236@users.noreply.github.com>
Co-authored-by: Abhishek Pathania <theabhishekpathania@gmail.com>
Co-authored-by: xykera <akrai9554@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: tcodehuber <tcodehuber@gmail.com>
Co-authored-by: LiangDai-Mars <dd574852610@gmail.com>
Co-authored-by: dailiang <dailiang@bytedance.com>
Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com>
Co-authored-by: baiyangtx <xiangnebula@163.com>
Co-authored-by: big face cat <731030576@qq.com>
Co-authored-by: davedwwang <iverson89w@163.com>
Co-authored-by: ConradJam <jam.gzczy@gmail.com>
Co-authored-by: Xu Bai <tocreationbai@gmail.com>
Co-authored-by: Jzjsnow <snow.jiangzj@gmail.com>
Co-authored-by: leosanqing <liurongtong001@qq.com>
Co-authored-by: leosanqing <stormleo@qq.com>
Co-authored-by: Sebb <sebbASF@users.noreply.github.com>
Co-authored-by: davedwwang <davedwwang@tencent.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:ams-server Ams server module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Save the last completion time of the executor cleanup

3 participants