[feature](iceberg) Support Partition Evolution DDL for Iceberg Tables#57972
Merged
morningman merged 13 commits intoapache:masterfrom Nov 26, 2025
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
suxiaogang223
force-pushed
the
iceberg_partition_evolution_ddl
branch
from
3c485cb to
7d998d1
Compare
Contributor
Author
|
run buildall |
zddr
reviewed
Nov 13, 2025
Contributor
zddr
left a comment
zddr
left a comment
There was a problem hiding this comment.
Reference regression-test/suites/mtmv_p0/test_iceberg_mtmv.groovy to add some partition evolution cases to validate the behavior of MTMV.
For example:
- Without changing the partition column, but switching from day to year, should be allowed. The materialized view should refresh normally and generate correct partitions.
- Changing the partition column from c1 to c2 should cause the materialized view refresh to fail.
- Switching from year to identity should cause the materialized view refresh to fail.
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34247 ms |
TPC-DS: Total hot run time: 188470 ms |
ClickBench: Total hot run time: 27.43 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34489 ms |
TPC-DS: Total hot run time: 187845 ms |
ClickBench: Total hot run time: 27.51 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
suxiaogang223
force-pushed
the
iceberg_partition_evolution_ddl
branch
2 times, most recently
from
57da735 to
26b5aef
Compare
suxiaogang223
force-pushed
the
iceberg_partition_evolution_ddl
branch
from
26b5aef to
f2cf736
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34708 ms |
TPC-DS: Total hot run time: 187048 ms |
There was a problem hiding this comment.
Pull request overview
This PR implements support for partition evolution in Apache Iceberg tables, enabling users to dynamically modify table partition strategies through DDL operations (ADD/DROP/REPLACE PARTITION KEY) without rewriting data files. The implementation follows Spark SQL syntax for consistency and maintains backward compatibility with historical data.
Key Changes:
- Added three new ALTER TABLE operations for partition field management (ADD/DROP/REPLACE)
- Extended grammar with partition transform expressions supporting bucket, truncate, year, month, day, hour, and identity transforms
- Implemented Iceberg-specific metadata operations through IcebergMetadataOps
- Added comprehensive test coverage including DDL tests, query/write tests, and MTMV compatibility tests
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| DorisParser.g4 | Added grammar rules for ADD/DROP/REPLACE PARTITION KEY with partition transform expressions |
| LogicalPlanBuilder.java | Implemented parser logic to extract partition field information and create operation objects |
| Add/Drop/ReplacePartitionFieldOp.java | Nereids operation classes for partition evolution with validation and SQL generation |
| Add/Drop/ReplacePartitionFieldClause.java | Analysis clause classes for legacy planner compatibility |
| AlterTableCommand.java | Extended to support partition field operations for external tables |
| Alter.java | Added validation to reject partition field operations on internal tables and handle external table operations |
| IcebergMetadataOps.java | Core implementation using Iceberg's UpdatePartitionSpec API with transform support |
| IcebergExternalCatalog.java | Added public methods to invoke partition evolution operations |
| IcebergExternalTable.java | Removed caching mechanism from isValidRelatedTable() to handle partition evolution |
| AlterOpType.java | Added three new operation types for partition evolution |
| test_iceberg_partition_evolution_*.groovy | Comprehensive test suites covering DDL, query/write, and MTMV scenarios |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
Outdated
Show resolved
Hide resolved
...rc/main/java/org/apache/doris/nereids/trees/plans/commands/info/ReplacePartitionFieldOp.java
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/analysis/AddPartitionFieldClause.java
Show resolved
Hide resolved
morningman
reviewed
Nov 22, 2025
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/IcebergDlaTable.java
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34838 ms |
TPC-DS: Total hot run time: 187181 ms |
ClickBench: Total hot run time: 27.48 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
zy-kkk
approved these changes
Nov 26, 2025
suxiaogang223
added a commit
to suxiaogang223/doris
that referenced
this pull request
Nov 30, 2025
…apache#57972) This PR implements support for partition evolution in Iceberg tables, allowing users to dynamically modify table partition strategies without rewriting data files. This is a metadata-only operation that maintains multiple partition spec versions. Apache Iceberg supports partition evolution, which enables changing partition strategies on existing tables without data migration. Doris, as a query engine for Iceberg, needs to support SQL syntax for partition evolution operations to provide users with flexible partition management. - **Add Partition Field**: Add new partition field to existing partition specifications - **Drop Partition Field**: Remove partition field from existing partition specifications - **Replace Partition Field**:Replace partition field from existing partition specifications with new partition field 1. **Metadata-only operation**: Partition evolution only updates table metadata, no data files are rewritten 2. **Backward compatibility**: Historical data retains original partition specs, new data uses new partition specs 3. **Syntax compatibility**: Follows Spark SQL ALTER TABLE syntax for consistency ```sql -- use optional AS keyword to specify a custom name for the partition field ALTER TABLE table_name ADD PARTITION KEY partition_transform [AS key_name]; -- example ALTER TABLE prod.db.sample ADD PARTITION KEY bucket(16, id); ALTER TABLE prod.db.sample ADD PARTITION KEY truncate(4, data); ALTER TABLE prod.db.sample ADD PARTITION KEY year(ts); -- use optional AS keyword to specify a custom name for the partition field ALTER TABLE prod.db.sample ADD PARTITION KEY bucket(16, id) AS shard; ``` ```sql ALTER TABLE table_name DROP PARTITION KEY partition_transform｜key_name; -- example ALTER TABLE prod.db.sample DROP PARTITION KEY catalog; ALTER TABLE prod.db.sample DROP PARTITION KEY bucket(16, id); ALTER TABLE prod.db.sample DROP PARTITION KEY truncate(4, data); ALTER TABLE prod.db.sample DROP PARTITION KEY year(ts); ALTER TABLE prod.db.sample DROP PARTITION KEY shard; ``` ```sql -- use optional AS keyword to specify a custom name for the partition field ALTER TABLE table_name REPLACE PARTITION KEY key_name WITH partition_transform [AS key_name]; -- example ALTER TABLE prod.db.sample REPLACE PARTITION KEY ts_day WITH day(ts); -- use optional AS keyword to specify a custom name for the new partition field ALTER TABLE prod.db.sample REPLACE PARTITION KEY ts_day WITH day(ts) AS day_of_ts; ``` | Transform | Syntax | Example | |-----------|--------|---------| | bucket | `bucket(N, column)` | `bucket(16, id)` | | truncate | `truncate(N, column)` | `truncate(10, name)` | | year | `year(column)` | `year(ts)` | | month | `month(column)` | `month(ts)` | | day | `day(column)` | `day(ts)` | | hour | `hour(column)` | `hour(ts)` | | identity | `column` | `category` |
nagisa-kunhah
pushed a commit
to nagisa-kunhah/doris
that referenced
this pull request
Dec 14, 2025
…apache#57972) ### What problem does this PR solve? ## Summary This PR implements support for partition evolution in Iceberg tables, allowing users to dynamically modify table partition strategies without rewriting data files. This is a metadata-only operation that maintains multiple partition spec versions. ## Background Apache Iceberg supports partition evolution, which enables changing partition strategies on existing tables without data migration. Doris, as a query engine for Iceberg, needs to support SQL syntax for partition evolution operations to provide users with flexible partition management. ## Features ### Core Functionality - **Add Partition Field**: Add new partition field to existing partition specifications - **Drop Partition Field**: Remove partition field from existing partition specifications - **Replace Partition Field**:Replace partition field from existing partition specifications with new partition field ### Design Principles 1. **Metadata-only operation**: Partition evolution only updates table metadata, no data files are rewritten 2. **Backward compatibility**: Historical data retains original partition specs, new data uses new partition specs 3. **Syntax compatibility**: Follows Spark SQL ALTER TABLE syntax for consistency ## Syntax ### Add Partition Field ```sql -- use optional AS keyword to specify a custom name for the partition field ALTER TABLE table_name ADD PARTITION KEY partition_transform [AS key_name]; -- example ALTER TABLE prod.db.sample ADD PARTITION KEY bucket(16, id); ALTER TABLE prod.db.sample ADD PARTITION KEY truncate(4, data); ALTER TABLE prod.db.sample ADD PARTITION KEY year(ts); -- use optional AS keyword to specify a custom name for the partition field ALTER TABLE prod.db.sample ADD PARTITION KEY bucket(16, id) AS shard; ``` ### Drop Partition Field ```sql ALTER TABLE table_name DROP PARTITION KEY partition_transform｜key_name; -- example ALTER TABLE prod.db.sample DROP PARTITION KEY catalog; ALTER TABLE prod.db.sample DROP PARTITION KEY bucket(16, id); ALTER TABLE prod.db.sample DROP PARTITION KEY truncate(4, data); ALTER TABLE prod.db.sample DROP PARTITION KEY year(ts); ALTER TABLE prod.db.sample DROP PARTITION KEY shard; ``` ### Replace Partition Field ```sql -- use optional AS keyword to specify a custom name for the partition field ALTER TABLE table_name REPLACE PARTITION KEY key_name WITH partition_transform [AS key_name]; -- example ALTER TABLE prod.db.sample REPLACE PARTITION KEY ts_day WITH day(ts); -- use optional AS keyword to specify a custom name for the new partition field ALTER TABLE prod.db.sample REPLACE PARTITION KEY ts_day WITH day(ts) AS day_of_ts; ``` ### Supported Partition Transforms | Transform | Syntax | Example | |-----------|--------|---------| | bucket | `bucket(N, column)` | `bucket(16, id)` | | truncate | `truncate(N, column)` | `truncate(10, name)` | | year | `year(column)` | `year(ts)` | | month | `month(column)` | `month(ts)` | | day | `day(column)` | `day(ts)` | | hour | `hour(column)` | `hour(ts)` | | identity | `column` | `category` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Summary
This PR implements support for partition evolution in Iceberg tables, allowing users to dynamically modify table partition strategies without rewriting data files. This is a metadata-only operation that maintains multiple partition spec versions.
Background
Apache Iceberg supports partition evolution, which enables changing partition strategies on existing tables without data migration. Doris, as a query engine for Iceberg, needs to support SQL syntax for partition evolution operations to provide users with flexible partition management.
Features
Core Functionality
Design Principles
Syntax
Add Partition Field
Drop Partition Field
Replace Partition Field
Supported Partition Transforms
bucket(N, column)bucket(16, id)truncate(N, column)truncate(10, name)year(column)year(ts)month(column)month(ts)day(column)day(ts)hour(column)hour(ts)columncategoryRelease note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)