[lake] Tolerate Paimon lake table existent with different insignificant options#1995
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements tolerance for Paimon lake tables that exist with different insignificant options, addressing issue #1994. The main goal is to allow Fluss to work with existing Paimon tables that have additional options set, as long as those options don't conflict with Fluss-managed settings.
Key Changes
- Introduced new validation logic that distinguishes between "changeable" (insignificant) and "unchangeable" options when comparing existing Paimon tables with new Fluss table schemas
- Added authorization checks to internal-only TabletService methods to ensure they can only be called by the Coordinator
- Refactored schema validation code from
PaimonLakeCataloginto a new dedicated utility classPaimonTableValidation
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| fluss-server/src/main/java/org/apache/fluss/server/tablet/TabletService.java | Added authorization checks to 6 internal-only RPC methods (notifyLeaderAndIsr, updateMetadata, stopReplica, notifyRemoteLogOffsets, notifyKvSnapshotOffset, notifyLakeTableOffset) to prevent external invocation |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/utils/PaimonTableValidation.java | New utility class for validating Paimon schema compatibility; extracts Paimon config options via reflection and implements logic to filter out changeable options when comparing schemas |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/PaimonLakeCatalog.java | Refactored to delegate schema validation to PaimonTableValidation; removed inline validation methods and unnecessary imports |
| fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/LakeEnabledTableCreateITCase.java | Added comprehensive test cases to verify the new option tolerance behavior, including tests for insignificant options, Fluss options, and Paimon options |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
luoyuxia
reviewed
Nov 18, 2025
Contributor
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
@LiebingYu Thanks for your pr. Left minor comment. PTAL
| removeChangeableOptions(newOptions); | ||
|
|
||
| // ignore the existing options that are not in new options | ||
| existingOptions.entrySet().removeIf(entry -> !newOptions.containsKey(entry.getKey())); |
Contributor
There was a problem hiding this comment.
why need this line code? Could you please give me an example?
Contributor
Author
There was a problem hiding this comment.
For example, the data lake manager may add a default value for snapshot.num-retained.min.
- user create a Fluss table with lake enable, say
t1, Fluss will call Paimon catalog to create a Paimon table withsnapshot.num-retained.min = 2; - user drop Fluss table
t1, the Paimon table should still exist; - user create
t1with same DDL again (nopaimon.snapshot.num-retained.minoption), we can ignoresnapshot.num-retained.minhere, let it keep the default value filled by the first creation.
| SchemaChange schemaChange3 = SchemaChange.setOption("fluss.k1", "v1"); | ||
| paimonCatalog.alterTable(paimonTablePath, Collections.singletonList(schemaChange3), false); | ||
|
|
||
| // add a new Paimon option (not specified in the Fluss table) to Paimon table will be ok |
Contributor
There was a problem hiding this comment.
shouldn't modify a Paimon option also throw exception? Imagine you modify the partition bucket key.
Contributor
Author
There was a problem hiding this comment.
These lines only change the fluss.k1 back to v1. Otherwise the following code cannot work.
LiebingYu
force-pushed
the
lake-paimon-options
branch
from
137836e to
df0f19c
Compare
Ugbot
pushed a commit
to Ugbot/fluss
that referenced
this pull request
Apr 26, 2026
…gnificant options (apache#1995)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #1994
Brief change log
Tests
API and Format
Documentation