Repository navigation
[fix](compaction) Back off repeated compaction failures - #69920
Open
YenchangChan wants to merge 2 commits into
Open
YenchangChan wants to merge 2 commits into
YenchangChan wants to merge 2 commits into
Conversation
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Link libsframe when available because binutils 2.40 moves SFrame symbols out of libbfd. ### Release note None ### Check List (For Author) - Test: No need to test (CMake feature detection only) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#66299 Related PR: None Problem Summary: Add capped exponential backoff for consecutive base and cumulative compaction failures, and reset the failure count after a successful compaction. ### Release note Add capped exponential backoff for repeatedly failing compaction tasks. ### Check List (For Author) - Test: Unit Test (TestTablet.compaction_failure_backoff) - Behavior changed: Yes (delay retries after consecutive compaction failures) - Does this need documentation: No
YenchangChan
requested review from
gavinchou,
luwei16 and
yiguolei
as code owners
November 1, 2026 01:23
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #66299
Related PR: None
Problem Summary:
When a tablet repeatedly fails compaction, the current scheduler retries it after the fixed
tablet_sched_delay_time_msdelay. A permanently failing tablet can therefore be selected every few seconds and continuously consume compaction threads and storage I/O without making progress.This implementation follows the analysis and proposed direction in #66299. Thanks to @furquan39 for the detailed investigation and production evidence.
This PR:
tablet_sched_delay_time_ms;compaction_failure_backoff_max_second, which defaults to 300 seconds;With the default configuration, retry delays increase as follows:
5s -> 10s -> 20s -> 40s -> ... -> 300sThe existing behavior for tablets without compaction failures remains unchanged.
This PR focuses on bounded retry backoff. It does not introduce a quarantine state or new metrics, which can be discussed separately based on operational needs and feedback.
Release note
Add capped exponential backoff for tablets with consecutive compaction failures.
Check List (For Author)
Test
TestTablet.compaction_failure_backoffBehavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)