branch-3.1: [fix](move-memtable) fix segment number mismatch for erroneously skipped segments #55092#55471
Merged
morrySnow merged 1 commit intobranch-3.1from Sep 4, 2025
Merged
Conversation
…ped segments (#55092) ### What problem does this PR solve? Fix segment number mismatch caused by erroneously skipped segments during concurrent incremental open on auto-partitioned table： #### Problem During concurrent incremental open on an auto-partitioned table, one sink may incorrectly assume that stream opened by another sink have already been opened and begin writing data while those segments are still being opened. This leads to some segments being silently skipped and results in a segment number mismatch. For example(two instances, 4 BEs: a, b, c, d): | Time | Event | | ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | t0 | `sink1` and `sink2` start incremental open for BEs **a, b, c, d**. | | t1 | `sink1` adds **a, b, c** to `_load_stream_map` and initiates open. | | t2 | `sink2` adds **d** to `_load_stream_map` and initiates open. | | t3 | `sink1` completes open for **a** and **b**; **c** is still in progress. | | t4 | `sink2` successfully opens **d**, assumes **a, b, c** are **all** ready, and starts writing. Because **c** is not yet fully open, its segments are skipped, causing the mismatch. | #### Expected behavior A sink must wait until all stream it depends on are fully opened before starting any write. #### Proposed fix All sinks open the full set of streams (a, b, c, d) instead of a partial subset. Lock on each stream guarantees that: - Duplicate open attempts are prevented:only the first sink performs the actual open; subsequent sinks wait until the open is complete. - Expected behavior is preserved:every sink waits until all streams are fully opened before starting any write, eliminating skipped segments and the resulting segment-number mismatch. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
TPC-H: Total hot run time: 32730 ms |
TPC-DS: Total hot run time: 192710 ms |
ClickBench: Total hot run time: 28.28 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
morrySnow
approved these changes
Sep 4, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #55092