[AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor - #3776

Jzjsnow · 2025-09-09T17:11:37Z

Why are the changes needed?

Close #3775.

Brief change log

Add support for MSE based refresh event:

Support for calculating partition filesize mean square error based on the loaded metadata.
Filter partitions need to be optimized based on threshold and trigger pendingInput evaluation if necessary.

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

xxubai · 2025-10-27T07:56:13Z

Can we move forward with this feature now? @Jzjsnow @klion26

Jzjsnow · 2025-10-31T08:03:36Z

Can we move forward with this feature now? @Jzjsnow @klion26

Sure, I've updated the branch and added the new evaluation criteria discussed earlier (see Step 1 for details).

The current conditions for triggering pendingInput evaluation based on metrics are as follows:
Step 1: If the condition delete file=0 && avg file size > target size * ratio is met, the evaluation is considered unnecessary and will be skipped.
Step 2: Calculate detailed attributes for each partition in the table, including the sum of squared errors for file sizes. If this exceeds the file size tolerance threshold, the pendingInput requires evaluation.

Note that this update now supports MIX_ICEBERG tables, whereas previously only ICEBERG format was supported.

Please take a look when you are free. Looking forward to your feedback! @xxubai @zhoujinsong @klion26

xxubai · 2025-10-31T09:27:29Z

+      "self-optimizing.evaluation.average-file-size.tolerance"; // the minimum tolerance value for
+  // the average
+  // partition file size (between 0 and (self-optimizing.target-size))
+  public static final MemorySize SELF_OPTIMIZING_EVALUATION_AVERAGE_FILE_SIZE_TOLERANCE_DEFAULT =


Can we use byte size to unify the file size unit?

Thanks for the suggestion. I have updated the variable to self-optimising.evaluation.file-size.mse-tolerance and specified bytes as the unit. This renders the variable name more clearly legible and aligns the unit with other size parameters.

xxubai · 2025-10-31T09:40:28Z

@@ -451,19 +462,132 @@ public List<PartitionBaseInfo> getTablePartitions(AmoroTable<?> amoroTable) {
        getTableFilesInternal(amoroTable, null, null);
    try {


Can use try catch with resource to close the io automaticly

Suggested change

try {

try (CloseableIterable<PartitionFileBaseInfo> tableFiles

= getTableFilesInternal(amoroTable, null, null)) {

for (PartitionFileBaseInfo fileInfo : tableFiles) {

refreshPartitionBasicInfo(fileInfo, partitionBaseInfoHashMap);

}

} catch (IOException e) {

LOG.warn("Failed to close the manifest reader.", e);

}

Sure, I’ve optimized the logic in MixedAndIcebergTableDescriptor.getTablePartitions().

xxubai · 2025-10-31T09:47:10Z

+      MixedTable table, long minTargetSize) {
+    Map<String, PartitionBaseInfo> partitionBaseInfoHashMap = new HashMap<>();
+    CloseableIterable<PartitionFileBaseInfo> tableFiles = getTableFilesInternal(table, null, null);
+    try {


You can simply use a try-with-resources statement.

xxubai · 2025-10-31T09:52:30Z

+      return true;
+    }
+
+    ExecutorService executorService = ThreadPools.getWorkerPool();


Use a dedicated thread pool to avoid thread congestion.

Suggested change

ExecutorService executorService = ThreadPools.getWorkerPool();

ExecutorService executorService = IcebergThreadPools.getPlanningExecutor();

The latest commit removes the separate thread pool for manifest file MSE calculation since we now handle this during the evaluation scan phase instead.

xxubai · 2025-10-31T10:18:23Z

+    return getTableFilesInternal(mixedTable, partition, specId);
+  }
+
+  private CloseableIterable<PartitionFileBaseInfo> getTableFilesInternal(


If I understand correctly, after the event-triggered evaluation, a full table scan is performed to collect partition information, which can be very expensive (especially for large tables with hundreds of thousands of files). Perhaps we can optimize this part when upgrading the Iceberg version and introducing PartitionStatistics.

You’re absolutely right. The latest commit moves partition file size MSE calculation to the evaluation scan phase, eliminating the need for an extra table scan. We’ll further optimize this with PartitionStatistics in future Iceberg upgrades to potentially avoid file-by-file traversal.

xxubai · 2025-10-31T10:23:46Z

+package org.apache.amoro.table;
+
+/** Detailed table partition properties list. */
+public class TablePartitionDetailProperties {


Can we simply this name? such as PartitionSummaryProperties

This class has been removed in the latest commit—we now directly reuse similar variables from the evaluator instead.

xxubai · 2025-10-31T10:32:52Z

              && lastOptimizedSnapshotId != defaultTableRuntime.getCurrentSnapshotId())) {
-        tryEvaluatingPendingInput(defaultTableRuntime, mixedTable);
+        if (!defaultTableRuntime.getOptimizingConfig().isEventBasedTriggerEnabled()
+            || MetricBasedRefreshEvent.isEvaluatingPendingInputNecessary(


Will this cause an additional full table scan compared to before?
In addition, we should also check whether optimization is enabled, so it would be better to combine this with tryEvaluatingPendingInput to avoid extra overhead.

Good call! In the latest commit, I've optimized the process to eliminate extra full table scans:

Before tryEvaluatingPendingInput, we only judge based on the table's average file size and number of delete files.

During tryEvaluatingPendingInput, MSE metrics are then calculated during the table scan.

We've also checked the metric-based trigger enabled in the RuntimeRefeshExecutor at the beginning,

Jzjsnow · 2025-11-20T08:28:28Z

In the latest commit, we've revamped the logic of EventBasedTrigger with key adjustments and new configurations:

The EventBasedTrigger now includes two key parameters:

FallbackInterval: The minimum interval for executing the original tryEvaluatingPendingInput logic. It prevents false positives or missed triggers from metadata metric-driven evaluation. Defaults to -1 (disabled); enabled when set to >=0.
MseTolerance: The tolerance threshold for partition file size MSE (default: 0). Partitions with actual MSE below this threshold are considered unnecessary for optimization.

When enabled, the flow now:

Determine if tryEvaluatingPendingInput needs to run:
- Check if the FallbackInterval is met to trigger tryEvaluatingPendingInput directly.
- Skip evaluation for empty tables.
- Skip if the condition delete file count = 0 && avg file size > target size * ratio is satisfied (no need for pending input evaluation).
Execute tryEvaluatingPendingInput if necessary:
- Use the existing scan logic to retrieve partition file information.
- Judge if each partition requires pending status: if the MSE threshold is met, further determine the optimization type (minor/major/full).
- Update pendingInput related information.

Please take a look where you are free! @xxubai @klion26

codecov-commenter · 2025-11-21T08:45:45Z

Codecov Report

❌ Patch coverage is 0% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 4.77%. Comparing base (cbdc517) to head (2fb2dc7).
⚠️ Report is 15 commits behind head on master.

Files with missing lines	Patch %	Lines
...izing/evaluation/MetadataBasedEvaluationEvent.java	0.00%	29 Missing ⚠️
...moro/optimizing/plan/CommonPartitionEvaluator.java	0.00%	27 Missing ⚠️
.../evaluation/MixedAndIcebergTableStatsProvider.java	0.00%	17 Missing ⚠️
...java/org/apache/amoro/config/OptimizingConfig.java	0.00%	15 Missing ⚠️
...moro/optimizing/evaluation/TableStatsProvider.java	0.00%	9 Missing ⚠️
...o/optimizing/plan/AbstractOptimizingEvaluator.java	0.00%	2 Missing ⚠️
...e/amoro/optimizing/plan/AbstractPartitionPlan.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #3776       +/-   ##
============================================
- Coverage     22.12%   4.77%   -17.36%     
+ Complexity     2461     471     -1990     
============================================
  Files           445     449        +4     
  Lines         40897   41048      +151     
  Branches       5767    5784       +17     
============================================
- Hits           9050    1958     -7092     
- Misses        31089   38896     +7807     
+ Partials        758     194      -564

Flag	Coverage Δ
trino	`4.77% <0.00%> (-17.36%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

turboFei

Great work, thanks

turboFei · 2025-11-27T22:39:01Z

+      -1; // event-based evaluation not in effect
+
+  public static final String SELF_OPTIMIZING_EVALUATION_FILE_SIZE_MSE_TOLERANCE =
+      "self-optimizing.evaluation.file-size.mse-tolerance";


maybe add some comments for the MSE abbreviation

Sure, I've added in the latest commit.

turboFei · 2025-11-27T22:44:59Z

  }

+  private void updateFileSizeSquaredErrorSum(DataFile dataFile) {
+    long diffSize = minTargetSize - Math.min(dataFile.fileSizeInBytes(), minTargetSize);


maybe add comments that only accumulates squared error for files smaller than minTargetSize.

Great point! I've updated the comments clarifying that the squared error accumulation is only applied to files with size smaller than minTargetSize.

turboFei · 2025-11-27T22:46:40Z


+  private void updateFileSizeSquaredErrorSum(DataFile dataFile) {
+    long diffSize = minTargetSize - Math.min(dataFile.fileSizeInBytes(), minTargetSize);
+    fileSizeSquaredErrorSum += diffSize * diffSize;


Is there a possibility of overflow here?

Thanks for catching this potential issue! I've fixed it by using Long.MAX_VALUE as the upper bound to prevent numeric overflow in the relevant logic.

turboFei · 2025-11-27T22:57:17Z

+
+      if (tableRuntime.getTableConfiguration().getOptimizingConfig().isEventBasedTriggerEnabled()
+          && !MetricBasedEvaluationEvent.isEvaluatingNecessary(
+              tableRuntime.getOptimizingConfig(), table, tableRuntime.getLastPlanTime())) {


what is the difference between

tableRuntime.getTableConfiguration().getOptimizingConfig() and tableRuntime.getOptimizingConfig(), it looks confuse.

Can we define variable likes OptimizingConfig config = tableRuntime.getOptimizingConfig() and reuse it to make it clear?

what is the difference between

tableRuntime.getTableConfiguration().getOptimizingConfig() and tableRuntime.getOptimizingConfig(), it looks confuse.

This is indeed puzzling. I see that the instances obtained by both methods are consistent. We may consider optimizing this in the future.

Anyway, an OptimizingConfig variable is defined and reused to improve code clarity.

turboFei · 2025-11-27T23:04:43Z

+
+  public static final String SELF_OPTIMIZING_EVALUATION_FILE_SIZE_MSE_TOLERANCE =
+      "self-optimizing.evaluation.file-size.mse-tolerance";
+  public static final long SELF_OPTIMIZING_EVALUATION_FILE_SIZE_MSE_TOLERANCE_DEFAULT = 0;


I saw that, now

if SELF_OPTIMIZING_EVALUATION_FALLBACK_INTERVAL_DEFAULT > 0, isEventBasedTriggerEnabled is true.

Should event-based evaluation be considered "enabled" if fallbackInterval ≥ 0 but evaluationMseTolerance == 0.

If not, can we give a default value for mse-tolerance or give some suggestions to determine the mse tolerance?

Thanks for your thoughtful question! When fallbackInterval ≥ 0 and evaluationMseTolerance == 0, the MSE metric is not actually used for evaluation, but the pre-planning judgment logic is still activated.

This logic includes two key steps which is in MetricBasedEvaluationEvent#isEvaluatingNecessary():
a. Skip empty tables;
b. Skip the pending input evaluation process if the conditions delete file=0 && avg file size > target size * ratio are both met.

turboFei · 2025-11-27T23:15:00Z

+      OptimizingConfig config, MixedTable table, long lastPlanTime) {
+    if (table.format() != TableFormat.ICEBERG && table.format() != TableFormat.MIXED_ICEBERG) {
+      logger.debug(
+          "MetricBasedRefreshEvent only support ICEBERG/MIXED_ICEBERG tables. Always return true for other table formats.");


is it by design?

MetricBasedRefreshEvent or MetricBasedEvaluationEvent

Thanks for pointing this out! This is indeed a typo in the log message and is fixed to align with the class name.

turboFei

LGTM, cc @xxubai can we move forward?

xxubai · 2025-12-12T07:30:17Z

+    // positives or
+    // missed triggers based on metadata metric-driven evaluation
+    if (isReachFallbackInterval(config, lastPlanTime)) {
+      logger.info("Maximum interval for evaluating table {} has reached.", table.id());


Set the log level to DEBUG. The evaluation process will be invoked frequently, generating a significant amount of log information

Sure, all the log levels have been modified

xxubai · 2025-12-12T09:18:57Z

  @Override
  public boolean isNecessary() {
    if (necessary == null) {
+      long lastPlanTime = Math.max(lastMinorOptimizingTime, lastFullOptimizingTime);


Lack last major optimizing time?

In the latest commit, the lastMajorOptimizingTime has been added to the partition evaluator alongside lastMinorOptimizingTime and lastFullOptimizingTime, enabling more comprehensive evaluations

xxubai · 2025-12-12T09:22:41Z

+
+import java.util.Map;
+
+public class MetricBasedEvaluationEvent {


Why is it named MetricBasedEvaluationEvent when it doesn’t receive any metric events? Additionally, it is only bound to the Iceberg/Mixed Iceberg format, making the coupling too strong.

This naming might be misleading, so in the latest commit, I renamed it to MetadataBasedEvaluationEvent as we use the table's file statistics to trigger evaluation.

The reason for supporting only Iceberg/Mixed formats is that table file statistics for these formats can be directly retrieved from the table summary without planning, whereas other formats cannot. However, I've abstracted a TableStatsProvider interface, which may enable support for other formats in the future. PTAL

xxubai · 2025-12-16T11:19:08Z

LGTM. Also need to fix the unit tests

…timeRefreshExecutor

xxubai · 2025-12-22T05:45:24Z

cc @klion26 @zhoujinsong @baiyangtx @Aireed @zhongqishang do you have any comments?

klion26

LGTM, thanks for the contribution.

zhoujinsong · 2025-12-22T08:47:13Z

Thanks for the great work! @Jzjsnow
Thanks for the review! @xxubai @klion26 @turboFei

…bleRuntimeRefreshExecutor (apache#3776) [AMORO-3775] Add support for metadata-based refresh event in TableRuntimeRefreshExecutor

@turboFei

…lave mode is enabled. (#3846) * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [AMORO-3850] Fix openapi sdk build and refine the package name (#3847) * rename the sdk pkg * version and artifactId * [HotFix] Change the uri configuration name in terminal (#3844) Change the uri configuration in terminal * [AMORO-3851] Separate spark-3.3 and spark-3.5 modules (#3848) * separate * java17 * build * GA * jdk17 tests * fix conflicts * <java.source.version>17</java.source.version> * JDK8 + Spark-3.5 * revert JDK17 * [AMORO-3857] Make the metadata file name conform to the Iceberg spec (#3858) Each version of table metadata is stored in a metadata folder under the table’s base location using a naming scheme that includes a version and UUID: <V>-<random-uuid>.metadata.json. * [AMORO-3850][FOLLOWUP] Add GA for OpenAPI SDK (#3870) [AMORO-3850][FOLLOWUP] Add GA for openAPI SDK * [AMORO-3864] Bump spark 3.5 version to 3.5.7 (#3860) Bump spark 3.5 version to 3.5.7 * [AMORO-3852][BUILD] Enhance build system (#3849) * Build project with fixed version check-latest: false GA * docs * shell * [AMORO-3851][FOLLOWUP] Using scala.binary.version for Spark dependency artifactId (#3861) Using scala.binary.version for dependency artifactId * [AMORO-3851][FOLLOWUP] Fix and simplify spark versions management (#3874) * Fix iceberg-spark artifactId in amoro-mixed-spark-3-common * fix ut conflicts * [AMORO-3863] Add script to reformat code (#3862) * format * spark profiles * [AMORO-3852] Using Maven Wrapper (#3879) * save * 3.9.9 * mvnw * slash * maven 3.9.11 * maven wrapper 3.3.4 * [AMORO-3880] Make amoro-openapi-sdk standalone (#3881) * [AMORO-3880] Make amoro-openapi-sdk standalone * save * [AMORO-3891] Bump netty version to 4.1.128.Final to fix CVE (#3892) Bump netty version to 4.1.128.Final to fix CVE * [AMORO-3890]Bump flink versions from 1.16.2/1.17.1/1.20.0 to 1.16.3/1.17.2/1.20.3 (#3889) bump * [AMORO-3880][FOLLOWUP] Fix pull request paths for OpenAPI SDK CI (#3903) * Bump orc-core version from 1.8.3 to 1.9.7 (#3900) * [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 (#3884) * Bump Paimon 1.1.1 to 1.2.0 * [AMORO-3883] Bump Paimon 1.1.1 to 1.2.0 --------- Co-authored-by: Xu Bai <xuba@apache.org> * [AMORO-3863][FOLLOWUP] Support to reformat pom (#3882) Support to reformat pom format trino * [AMORO-3872] Support to customize basic authentication implementation (#3871) * support to customize basic auth" * save * docs * common * address comments * nit --------- Co-authored-by: Xu Bai <xuba@apache.org> * [AMORO-3907] Bump kubernetes-client version to 6.13.5 (#3908) Bump kubernetes-client version to 6.13.5 * [AMORO-3875] Support to build on Spark Scala-2.13 (#3878) * scala binary version save bin profile GA revert paimon ams api compatiblity fix flink scala style save save rewrite by scala save conflicts dependency save save scala paimon ignore paimon idea * nit * nit * save * revert ci change * [AMORO-3875][FOLLOWUP] Fix optimizer-spark docker image build due to artifactId change (#3912) * fix save * test * Revert "test" This reverts commit 6c38dbd. * [AMORO-3863][FOLLOWUP] Enable format-mixed-format-trino profile on JDK17+ automatically (#3906) reformat * [AMORO-3848][FOLLOWUP] Prompt spark profile in docker build (#3904) Prompt spark profile in docker build * [AMORO-3918][INFRA] Add dependencies check CI (#3917) * [AMORO-3931] Exclude curator and zookeeper deps (#3932) * save deps exclude jline * nit * [AMORO-3945][DOCS] Add docs for REST API (#3944) docs combine * [AMORO-3873] Support Bearer/JWT authentication (#3905) * save * save * save * save * Save * asve * rename package * comments * basic * remove token from log * [AMORO-3938] change file_content_b64 column type to CLOB type (#3939) * [AMORO-3933] Fix Playground demo failure due `PartitionExpressionForMetastore class not found` (#3935) * Fix Playground demo failure due PartitionExpressionForMetastore class not found * Revert "Fix Playground demo failure due PartitionExpressionForMetastore class not found" This reverts commit fea6ddc. * add for runtime hive-exec --------- Co-authored-by: Xu Bai <xuba@apache.org> * Save the last completion time for each cleanup operation performed on each optimization table. (#3802) * Save the last completion time for each cleanup operation performed on each optimization table. # Conflicts: # amoro-ams/src/main/resources/mysql/upgrade.sql * Store the execution time of each cleanup operation for the optimization table in the table_runtime_state table. * fixup style --------- Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * nit: remove duplicate code (#3957) NIT: Remove duplicate code * [AMORO-3873][FOLLOWUP] Rename rest auth bearer type to JWT (#3953) * [AMORO-3968] Update the thrift api compile command to use amoro shaded thrift (#3967) * relocate generated thrift code to use amoro shaded thrift * docs * save * profile * [AMORO-3961] Filter null key and value for Configurations::toMap (#3962) filter null key value * nit: remove unused code (#3959) remove unused code * [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 (#3975) * [AMORO-3972][Core] Upgrade default Spark version from 3.3 to 3.5 This PR upgrades the default Spark version from 3.3 to 3.5 in the Amoro project. Changes: - pom.xml: Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5 - pom.xml (hadoop2 profile): Update spark.version from 3.3.4 to 3.5.7 and spark.major.version from 3.3 to 3.5 - docker/build.sh: Update SPARK_VERSION from 3.3.3 to 3.5.7 - docker/optimizer-spark/Dockerfile: Update ARG SPARK_VERSION from 3.3.3 to 3.5.7 - .github/workflows/docker-images.yml: Update Spark optimizer matrix from 3.3.3 to 3.5.7 - README.md: Update Spark optimizer default version documentation Closes #3972 * Keep Spark 3.3 as default for hadoop2 profile Address review feedback from @turboFei: For hadoop2 profile, keep using the legacy Spark 3.3 by default for better compatibility with Hadoop 2.x. * Fix Spark version parameter name in README.md Address review feedback from @turboFei: Change -Dspark-optimizer.spark-version to -Dspark.version This was missed in PR #3874. * [AMORO-3966][Helm] Support custom volumes and volumeMounts (#3965) [Improvement][Helm] Support custom volumes and volumeMounts * [AMORO-3970] Update README.md with correct Spark versions for Mixed format (#3978) * [AMORO-3977] Combine amoro-site to prevent docs loose sync (#3979) * clone site * remove invalid soft links * docs soft link * Update site/README.md with clear structure documentation - Reorganized documentation to show the versioned and non-versioned content structure - Provided a clear directory tree representation of the site structure - Updated instructions for running the documentation site locally - Added section on testing both sites together 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Update site/README.md with clear structure documentation - Fixed references to make it clear this is part of the main repository - Updated section titles to match the new organization - Clarified paths for versioned and non-versioned content - Simplified local development instructions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * github action * ignore * rat exclude site * test * test please * Revert "test please" This reverts commit 7c536fe. * label --------- Co-authored-by: Claude <noreply@anthropic.com> * [Improvement]: Disable verification for helm-unittest plugin installation (#3993) Disable verification for helm-unittest plugin installation * [AMORO_3990] Add package chart instructions to README (#3991) * [Feature] Introduce a new framework that supports scheduling for Formats and Processes (#3924) * process service poc * Refactor table process framework * Optimize and refactor the process service framework & support framework test case --------- Co-authored-by: dailiang <dailiang@bytedance.com> Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com> Co-authored-by: baiyangtx <xiangnebula@163.com> * [AMORO-3981] Automatic generate amoro configuration docs (#3982) * auto generate ams config * add auto generate notice * junit 5 * convertToString * Update CONTRIBUTING.md --------- Co-authored-by: big face cat <731030576@qq.com> * [AMORO-3981][FOLLOWUP] Refine the duration to string format (#3987) * [AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string * Revert "[AMORO-3981][FOLLOWUP] Refine duration type parse and convert to string" This reverts commit 877bcb9. * format duration with TimeUtils.formatWithHighestUnit * largest unit * [AMORO-1258] Support Zookeeper kerberos authentication (#3980) * [AMORO-1258] Support Zookeeper kerberos authentication * remove minkdc * Revert "remove minkdc" This reverts commit 466d632. * docs * [AMORO-3934] Manage com.fasterxml.jackson dependencies in dependencyManagement (#3936) * jackson version * deps * fasterxml.jackson.version --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-3866] Using shaded hadoop-client-api and hadoop-client-runtime for hadoop3 (#3983) hadoop client api and hadoop client runtime remove guava test replace exclude remove runtime * [AMORO-3632] Refine data expiration literal calculation for date type (#3964) * [Feature]: data-retention, add support for partition column type Date #3632 * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632]: data-retention, add support for partition column type Date * [AMORO-3632] Fix #3665: Prevent long overflow in Date expiration calculation and fix related tests --------- * [AMORO-3885] fix missing oss.endpoint for internal mixed_iceberg catalog (#3886) * fix missing oss.endpoint for internal mixed_iceberg catalog * fix missing oss.endpoint for internal mixed_iceberg catalog --------- Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> * [AMORO-2635] Enhance table partition files list performance (#4003) * [AMORO-2635] * UT * UT for fallback * address comments * TODO * [AMORO-3628] Add user logo wall on the home page. (#4010) * [AMORO-3628] Add users logo wall in the home page * update Co-authored-by: Claude * [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration (#4005) * [AMORO-3804] Skip RUNTIME_CONTEXT_CACHE for TableMetaStore with local configuration * return * test * configuration * create runtime context directly * remove unneeded tests * [AMORO-4011] Fix JUnit 4 tests skipped (#4017) * add back * test * revert test * deps * [AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor (#3776) [AMORO-3775] Add support for metadata-based refresh event in TableRuntimeRefreshExecutor * [AMORO-3998]support DATABASE HA SERVICE (#3997) support DATABASE HA SERVICE Co-authored-by: dailiang <dailiang@bytedance.com> * [Hotfix] Fix the deploy site GitHub workflow (#4020) * Fix deploy site GitHub workflow * Add publish information in the .asf.yaml * [Hotfix] Enable manual triggering for site deployment workflow (#4021) * [Hotfix]Refactor the build command in the `Publish Docker Image` GitHub workflow to reduce image size (#4023) Refactor the build command in the Publish DOcker Image GitHub workflow to reduce image size * [Hotfix] Fix typos (#4026) * [AMORO-3531] Drop support for java8 (#3899) drop support jdk8 Co-authored-by: Xu Bai <xuba@apache.org> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> * [AMORO-3940] Flink config load will cause ClassCastException when dir… (#3941) [AMORO-3940] Flink config load will cause ClassCastException when directly return result of Yaml load Co-authored-by: ConradJam <jam.gzczy@gmail.com> * [AMORO-3943] Shade all third party classes for spark runtime (#3942) * relocate * filter * [AMORO-4027] Bump shade plugin to fix compilation failure on JDK 17 (#4028) (cherry picked from commit 83e5272) * [AMORO-3973] Support spark3.4 for mixed format (#4013) * Copy the code form spark3.5 * Optimize code * Maintain consistency in the references to scala-compiler and scala-library across Spark 3.5, Spark 3.4, and Spark 3.3 * fixup --------- Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> Co-authored-by: Xu Bai <tocreationbai@gmail.com> * [Hotfix] Try to fix site-deploy permission problem (#4025) * [AMORO-4022] [Improvement]: AMS Iceberg maintainer moved to the amoro-iceberg module (#4024) [AMORO-4022] AMS Iceberg maintainer moved to the amoro-iceberg module (#4022) * [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996) [amoro-3994] Support Exposing AMS High Availability (HA) Status Co-authored-by: davedwwang <davedwwang@tencent.com> * [Feature]: Support LDAP Authentication for Dashboard Login (#4009) Co-authored-by: davedwwang <davedwwang@tencent.com> * [HotFix] Remove duplicate references to scala-library and scala-compiler (#4033) Remove duplicate references to scala-library and scala-compiler Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * [HotFix] Shade third party classes for spark-3.4 runtime (#4032) Shade third party classes for spark runtime 3.4 Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [AMORO-3998]support DATABASE HA SERVICE (#3997) support DATABASE HA SERVICE Co-authored-by: dailiang <dailiang@bytedance.com> * [AMORO-3994] Support Exposing AMS High Availability (HA) Status (#3996) [amoro-3994] Support Exposing AMS High Availability (HA) Status Co-authored-by: davedwwang <davedwwang@tencent.com> * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Use a new configuration item to control whether master & slave mode is enabled. #3845 * [Subtask]: Optimize the description information for use-master-slave-mode. --------- Co-authored-by: wardli <wardli@tencent.com> Co-authored-by: Fei Wang <fwang12@ebay.com> Co-authored-by: ZhouJinsong <zhoujinsong0505@163.com> Co-authored-by: yeatsliao <liaoyt66066@gmail.com> Co-authored-by: Xu Bai <xuba@apache.org> Co-authored-by: xuzifu666 <xuzifu666@gmail.com> Co-authored-by: simonsssu <barley0806@gmail.com> Co-authored-by: zhangwl9 <1298877813@qq.com> Co-authored-by: 张文领 <zhangwl9@chinatelecom.cn> Co-authored-by: zhan7236 <76658920+zhan7236@users.noreply.github.com> Co-authored-by: Abhishek Pathania <theabhishekpathania@gmail.com> Co-authored-by: xykera <akrai9554@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: tcodehuber <tcodehuber@gmail.com> Co-authored-by: LiangDai-Mars <dd574852610@gmail.com> Co-authored-by: dailiang <dailiang@bytedance.com> Co-authored-by: zhangyongxiang.alpha <zhangyongxiang.alpha@bytedance.com> Co-authored-by: baiyangtx <xiangnebula@163.com> Co-authored-by: big face cat <731030576@qq.com> Co-authored-by: davedwwang <iverson89w@163.com> Co-authored-by: ConradJam <jam.gzczy@gmail.com> Co-authored-by: Xu Bai <tocreationbai@gmail.com> Co-authored-by: Jzjsnow <snow.jiangzj@gmail.com> Co-authored-by: leosanqing <liurongtong001@qq.com> Co-authored-by: leosanqing <stormleo@qq.com> Co-authored-by: Sebb <sebbASF@users.noreply.github.com> Co-authored-by: davedwwang <davedwwang@tencent.com>

github-actions Bot added module:ams-server Ams server module module:common labels Sep 9, 2025

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 2 times, most recently from cd57764 to c8734fb Compare September 29, 2025 09:11

Jzjsnow changed the title ~~[AMORO-3775] Add support for pluggable refresh event in TableRuntimeRefreshExecutor~~ [AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor Sep 29, 2025

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 2 times, most recently from c352653 to f00825b Compare September 29, 2025 09:41

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 3 times, most recently from ab4b971 to 519d183 Compare October 31, 2025 07:50

xxubai reviewed Oct 31, 2025

View reviewed changes

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 5 times, most recently from 22dd931 to d40535b Compare November 19, 2025 03:56

klion26 force-pushed the add_support_for_pluggable_refresh_event branch from d40535b to 7808f03 Compare November 21, 2025 08:38

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 2 times, most recently from 082da43 to 737b63f Compare November 25, 2025 09:38

turboFei reviewed Nov 27, 2025

View reviewed changes

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 2 times, most recently from e3bddc2 to 3a8c2d7 Compare December 12, 2025 06:28

turboFei approved these changes Dec 12, 2025

View reviewed changes

xxubai reviewed Dec 12, 2025

View reviewed changes

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch from 3a8c2d7 to 22548f5 Compare December 16, 2025 10:03

github-actions Bot added the module:mixed-hive Hive moduel for Mixed Format label Dec 16, 2025

xxubai approved these changes Dec 16, 2025

View reviewed changes

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 4 times, most recently from 2fb2dc7 to 75d2622 Compare December 19, 2025 06:49

[AMORO-3775] Add support for metadata-based refresh event in TableRun…

ae607af

…timeRefreshExecutor

Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch from 75d2622 to ae607af Compare December 19, 2025 07:05

klion26 approved these changes Dec 22, 2025

View reviewed changes

zhoujinsong merged commit ab713b1 into apache:master Dec 22, 2025
8 checks passed

		@@ -451,19 +462,132 @@ public List<PartitionBaseInfo> getTablePartitions(AmoroTable<?> amoroTable) {
		getTableFilesInternal(amoroTable, null, null);
		try {

-    try {
+        try (CloseableIterable<PartitionFileBaseInfo> tableFiles
+                 = getTableFilesInternal(amoroTable, null, null)) {
+      for (PartitionFileBaseInfo fileInfo : tableFiles) {
+        refreshPartitionBasicInfo(fileInfo, partitionBaseInfoHashMap);
+      }
+    } catch (IOException e) {
+      LOG.warn("Failed to close the manifest reader.", e);
+    }

	ExecutorService executorService = ThreadPools.getWorkerPool();
	ExecutorService executorService = IcebergThreadPools.getPlanningExecutor();


		import java.util.Map;

		public class MetricBasedEvaluationEvent {

Uh oh!

Conversation

Jzjsnow commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Brief change log

How was this patch tested?

Documentation

Uh oh!

xxubai commented Oct 27, 2025

Uh oh!

Jzjsnow commented Oct 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jzjsnow Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jzjsnow commented Nov 20, 2025

Uh oh!

codecov-commenter commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

turboFei left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turboFei Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jzjsnow Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jzjsnow Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jzjsnow commented Sep 9, 2025 •

edited

Loading

Jzjsnow Nov 20, 2025 •

edited

Loading

codecov-commenter commented Nov 21, 2025 •

edited

Loading

turboFei Nov 27, 2025 •

edited

Loading

Jzjsnow Dec 9, 2025 •

edited

Loading

Jzjsnow Dec 9, 2025 •

edited

Loading