Support flexible partial update for routine load#60128

dataroaring · 2026-01-21T17:28:24Z

This feature adds support for UPDATE_FLEXIBLE_COLUMNS mode in routine load, allowing partial updates where the columns to update are determined dynamically from each JSON row's fields rather than being fixed upfront.

Key changes:

Add unique_key_update_mode property to routine load (UPSERT, UPDATE_FIXED_COLUMNS, UPDATE_FLEXIBLE_COLUMNS)
Validate flexible partial update constraints (JSON format only, no jsonpaths, no COLUMNS clause, no fuzzy_parse)
Support ALTER ROUTINE LOAD to change update mode
Add comprehensive regression tests for all scenarios
Add waitForTaskFinishMoW utility for MOW table tests

Constraints for flexible partial update:

Only JSON format supported
Cannot use jsonpaths (columns determined from JSON fields)
Cannot use COLUMNS clause
Cannot use fuzzy_parse
Table must be MOW with skip_bitmap column enabled
Table cannot have variant columns

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

This feature adds support for UPDATE_FLEXIBLE_COLUMNS mode in routine load, allowing partial updates where the columns to update are determined dynamically from each JSON row's fields rather than being fixed upfront. Key changes: - Add unique_key_update_mode property to routine load (UPSERT, UPDATE_FIXED_COLUMNS, UPDATE_FLEXIBLE_COLUMNS) - Validate flexible partial update constraints (JSON format only, no jsonpaths, no COLUMNS clause, no fuzzy_parse) - Support ALTER ROUTINE LOAD to change update mode - Add comprehensive regression tests for all scenarios - Add waitForTaskFinishMoW utility for MOW table tests Constraints for flexible partial update: - Only JSON format supported - Cannot use jsonpaths (columns determined from JSON fields) - Cannot use COLUMNS clause - Cannot use fuzzy_parse - Table must be MOW with skip_bitmap column enabled - Table cannot have variant columns

Thearas · 2026-01-21T17:28:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

dataroaring · 2026-01-21T17:28:48Z

run buildall

Copilot

Pull request overview

This PR adds support for flexible partial update mode (UPDATE_FLEXIBLE_COLUMNS) in routine load operations for Apache Doris. The feature enables partial updates where the columns to update are determined dynamically from each JSON row's fields, rather than being fixed upfront.

Changes:

Introduced unique_key_update_mode property supporting UPSERT, UPDATE_FIXED_COLUMNS, and UPDATE_FLEXIBLE_COLUMNS modes
Added validation constraints for flexible partial updates (JSON format only, MOW tables with skip_bitmap column, no jsonpaths/fuzzy_parse/COLUMNS clause)
Implemented ALTER ROUTINE LOAD support to change update modes
Added comprehensive regression tests covering all scenarios and edge cases
Introduced waitForTaskFinishMoW utility for testing MOW table operations

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
test_routine_load_partial_update_new_key_behavior.groovy	Updated error message to be more general (partial update vs partial_columns)
test_routine_load_flexible_partial_update.groovy	New comprehensive test suite with 21 test cases covering feature functionality and validation
test_routine_load_flexible_partial_update.out	Expected test output file for the new test suite
RoutineLoadTestUtils.groovy	Added waitForTaskFinishMoW method for MOW table testing with skip_delete_bitmap support
CreateRoutineLoadInfo.java	Added parsing, validation logic, and flexible partial update constraints
AlterRoutineLoadCommand.java	Added unique_key_update_mode to alterable properties
NereidsStreamLoadPlanner.java	Refactored validation to use centralized OlapTable.validateForFlexiblePartialUpdate
NereidsRoutineLoadTaskInfo.java	Updated constructor to accept uniqueKeyUpdateMode parameter
RoutineLoadJob.java	Added update mode tracking, backward compatibility, and ALTER validation logic
KafkaRoutineLoadJob.java	Updated to pass uniqueKeyUpdateMode to task info
OlapTable.java	Added centralized validateForFlexiblePartialUpdate method
RoutineLoadJobTest.java	Added comprehensive unit tests for parsing and validation logic

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-21T17:38:58Z

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

+            throw new UserException("Flexible partial update can only support table with skip bitmap hidden column."
+                    + " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()
+                    + " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");
+        }
+        if (!getEnableLightSchemaChange()) {
+            throw new UserException("Flexible partial update can only support table with light_schema_change enabled."
+                    + " But table " + getName() + "'s property light_schema_change is false");
+        }
+        if (hasVariantColumns()) {
+            throw new UserException("Flexible partial update can only support table without variant columns.");


The error message text has an inconsistency with the grammar. The message says "can only support table" but should say "only supports tables" (plural) for better grammar, or "can only support a table" (singular with article) for grammatical correctness.

Suggested change

throw new UserException("Flexible partial update can only support table with skip bitmap hidden column."

+ " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()

+ " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");

}

if (!getEnableLightSchemaChange()) {

throw new UserException("Flexible partial update can only support table with light_schema_change enabled."

+ " But table " + getName() + "'s property light_schema_change is false");

}

if (hasVariantColumns()) {

throw new UserException("Flexible partial update can only support table without variant columns.");

throw new UserException("Flexible partial update only supports tables with a skip bitmap hidden column."

+ " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()

+ " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");

}

if (!getEnableLightSchemaChange()) {

throw new UserException("Flexible partial update only supports tables with light_schema_change enabled."

+ " But table " + getName() + "'s property light_schema_change is false");

}

if (hasVariantColumns()) {

throw new UserException("Flexible partial update only supports tables without variant columns.");

Copilot · 2026-01-21T17:38:59Z

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

+            throw new UserException("Flexible partial update can only support table with skip bitmap hidden column."
+                    + " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()
+                    + " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");
+        }
+        if (!getEnableLightSchemaChange()) {
+            throw new UserException("Flexible partial update can only support table with light_schema_change enabled."
+                    + " But table " + getName() + "'s property light_schema_change is false");
+        }
+        if (hasVariantColumns()) {
+            throw new UserException("Flexible partial update can only support table without variant columns.");


The error message text has an inconsistency with the grammar. The message says "can only support table" but should say "only supports tables" (plural) for better grammar, or "can only support a table" (singular with article) for grammatical correctness.

Suggested change

throw new UserException("Flexible partial update can only support table with skip bitmap hidden column."

+ " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()

+ " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");

}

if (!getEnableLightSchemaChange()) {

throw new UserException("Flexible partial update can only support table with light_schema_change enabled."

+ " But table " + getName() + "'s property light_schema_change is false");

}

if (hasVariantColumns()) {

throw new UserException("Flexible partial update can only support table without variant columns.");

throw new UserException("Flexible partial update only supports tables with a skip bitmap hidden column."

+ " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()

+ " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");

}

if (!getEnableLightSchemaChange()) {

throw new UserException("Flexible partial update only supports tables with light_schema_change enabled."

+ " But table " + getName() + "'s property light_schema_change is false");

}

if (hasVariantColumns()) {

throw new UserException("Flexible partial update only supports tables without variant columns.");

Copilot · 2026-01-21T17:38:59Z

regression-test/suites/load_p0/routine_load/test_routine_load_flexible_partial_update.groovy

+                    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
+                );
+            """
+            exception "Flexible partial update can only support table with skip bitmap hidden column"


The error message text has an inconsistency with the grammar. The message says "can only support table" but should say "only supports tables" (plural) for better grammar, or "can only support a table" (singular with article) for grammatical correctness.

Suggested change

exception "Flexible partial update can only support table with skip bitmap hidden column"

exception "Flexible partial update only supports tables with skip bitmap hidden column"

Copilot · 2026-01-21T17:38:59Z

regression-test/suites/load_p0/routine_load/test_routine_load_flexible_partial_update.groovy

+                    "property.kafka_default_offsets" = "OFFSET_BEGINNING"
+                );
+            """
+            exception "Flexible partial update can only support table without variant columns"


The error message text has an inconsistency with the grammar. The message says "can only support table" but should say "only supports tables" (plural) for better grammar, or "can only support a table" (singular with article) for grammatical correctness.

Suggested change

exception "Flexible partial update can only support table without variant columns"

exception "Flexible partial update only supports tables without variant columns"

Copilot · 2026-01-21T17:39:00Z

regression-test/suites/load_p0/routine_load/test_routine_load_flexible_partial_update.groovy

+                        "unique_key_update_mode" = "UPDATE_FLEXIBLE_COLUMNS"
+                    );
+                """
+                exception "Flexible partial update can only support table with skip bitmap hidden column"


The error message text has an inconsistency with the grammar. The message says "can only support table" but should say "only supports tables" (plural) for better grammar, or "can only support a table" (singular with article) for grammatical correctness.

Suggested change

exception "Flexible partial update can only support table with skip bitmap hidden column"

exception "Flexible partial update can only support a table with skip bitmap hidden column"

Copilot · 2026-01-21T17:39:00Z

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

+            throw new UserException("Flexible partial update can only support table with skip bitmap hidden column."
+                    + " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()
+                    + " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");
+        }
+        if (!getEnableLightSchemaChange()) {
+            throw new UserException("Flexible partial update can only support table with light_schema_change enabled."
+                    + " But table " + getName() + "'s property light_schema_change is false");
+        }
+        if (hasVariantColumns()) {
+            throw new UserException("Flexible partial update can only support table without variant columns.");


The error message text has an inconsistency with the grammar. The message says "can only support table" but should say "only supports tables" (plural) for better grammar, or "can only support a table" (singular with article) for grammatical correctness.

Suggested change

throw new UserException("Flexible partial update can only support table with skip bitmap hidden column."

+ " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()

+ " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");

}

if (!getEnableLightSchemaChange()) {

throw new UserException("Flexible partial update can only support table with light_schema_change enabled."

+ " But table " + getName() + "'s property light_schema_change is false");

}

if (hasVariantColumns()) {

throw new UserException("Flexible partial update can only support table without variant columns.");

throw new UserException("Flexible partial update is only supported for tables with a skip bitmap hidden column."

+ " But table " + getName() + " doesn't have it. You can use `ALTER TABLE " + getName()

+ " ENABLE FEATURE \"UPDATE_FLEXIBLE_COLUMNS\";` to add it to the table.");

}

if (!getEnableLightSchemaChange()) {

throw new UserException("Flexible partial update is only supported for tables with light_schema_change enabled."

+ " But table " + getName() + "'s property light_schema_change is false");

}

if (hasVariantColumns()) {

throw new UserException("Flexible partial update is only supported for tables without variant columns.");

hello-stephen · 2026-01-21T23:49:25Z

FE UT Coverage Report

Increment line coverage 16.03% (21/131) 🎉
Increment coverage report
Complete coverage report

dataroaring requested a review from yiguolei as a code owner January 21, 2026 17:28

Copilot AI review requested due to automatic review settings January 21, 2026 17:28

Copilot started reviewing on behalf of dataroaring January 21, 2026 17:29 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

yiguolei approved these changes Feb 6, 2026

View reviewed changes

yiguolei merged commit 80283db into branch-4.0 Feb 6, 2026
32 of 41 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support flexible partial update for routine load#60128