[fix][broker] Trigger max read position callback for messages published during transaction buffer recovery - #26234
Merged
lhotari merged 1 commit intoJul 23, 2026
Conversation
…ed during transaction buffer recovery While TopicTransactionBuffer is recovering, syncMaxReadPositionForNormalPublish ignores normal publishes, and recoverComplete/noNeedToRecover advanced maxReadPosition by direct field assignment without triggering the maxReadPositionMovedForward callback. As a result, messages published during recovery never updated PersistentTopic.lastMaxReadPositionMovedForwardTimestamp, so ReplicatedSubscriptionsController.startNewSnapshot considered the topic to have no new data and never started a subscription snapshot until further traffic arrived. This made ReplicatedSubscriptionWithTransactionTest.testReplicatedSubscribeAndSwitchToStandbyCluster flaky: when all its messages were published before the transaction buffer recovery of the new topic completed, no snapshot could ever complete and the test timed out. Advance maxReadPosition through a helper that fires the callback when the position actually moved forward during recovery, and add a deterministic regression test that gates recovery on a controllable future. Assisted-by: Claude Code (Fable 5)
dao-jun
approved these changes
Jul 23, 2026
11 tasks
sandeep-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jul 31, 2026
…ed during transaction buffer recovery (apache#26234) (cherry picked from commit 520b0da)
sandeep-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jul 31, 2026
…ed during transaction buffer recovery (apache#26234) (cherry picked from commit 520b0da)
sandeep-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jul 31, 2026
…ed during transaction buffer recovery (apache#26234) (cherry picked from commit 520b0da)
nodece
pushed a commit
to ascentstream/pulsar
that referenced
this pull request
Aug 28, 2026
…ed during transaction buffer recovery (apache#26234) (cherry picked from commit 520b0da)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ReplicatedSubscriptionWithTransactionTest.testReplicatedSubscribeAndSwitchToStandbyClusterfails intermittently in CI with:Example failure: https://github.com/apache/pulsar/actions/runs/29992607875/job/89160931729 (a dependabot PR unrelated to this code path).
The captured test logs show that both clusters' replicators were connected and all test messages were replicated within ~1 second of the test starting, followed by total silence: no replicated-subscription snapshot round trip was ever attempted during the remaining ~9 seconds of the Awaitility window.
Root cause is in
TopicTransactionBuffer, not in the test:Initializingstate),syncMaxReadPositionForNormalPublish()ignores normal publishes:maxReadPositiondoes not move and themaxReadPositionMovedForwardcallback is not invoked.recoverComplete()/noNeedToRecover()catchmaxReadPositionup to the managed ledger's last confirmed entry by direct field assignment, without invoking themaxReadPositionMovedForwardcallback.As a result, messages published while the transaction buffer is recovering never update
PersistentTopic.lastMaxReadPositionMovedForwardTimestamp.ReplicatedSubscriptionsController.startNewSnapshot()uses that timestamp to decide whether the topic has new data; with the timestamp stuck at 0, every snapshot tick is skipped with "There is no new data in topic. Skipping snapshot creation." until some new message is published after recovery.In the flaky test, the whole publish burst races the transaction buffer recovery of the freshly created topic. When all messages land before recovery completes, no subscription snapshot can ever start (the test publishes nothing more while polling the assertion), so the test times out regardless of the Awaitility timeout. In production the condition self-heals on the next publish, which is why it mainly surfaces as a test flake, but until new traffic arrives a replicated-subscription snapshot is silently delayed the same way.
Modifications
TopicTransactionBuffer: extract aupdateMaxReadPositionAfterRecovery()helper used by bothrecoverComplete()andnoNeedToRecover(). It advancesmaxReadPositionto the last confirmed entry and invokes themaxReadPositionMovedForwardcallback if the position actually moved forward while recovery was in progress. When nothing was published during recovery, the position does not move and the callback is not invoked, preserving the existing "no snapshot without traffic" behavior (covered byReplicatedSubscriptionTest.testReplicationSnapshotStopWhenNoTraffic).TopicTransactionBufferRecoveryTest, a deterministic regression test that gates transaction buffer recovery on a controllable future (mockedAbortedTxnProcessor.recoverFromSnapshot()), publishes messages while recovery is pending, and asserts that completing the recovery (both with and without a snapshot to replay) moveslastMaxReadPositionMovedForwardTimestampforward. It also asserts the negative case: recovery of an idle topic must not move the timestamp.Verifying this change
This change added tests and can be verified as follows:
TopicTransactionBufferRecoveryTest.testMaxReadPositionMovedForwardForMessagesPublishedDuringRecoverydeterministically reproduces the bug: it fails without the production change and passes with it.TopicTransactionBufferRecoveryTest.testMaxReadPositionNotMovedForwardWhenNothingPublishedDuringRecoveryguards against spuriously triggering snapshots for idle topics.ReplicatedSubscriptionWithTransactionTest.testReplicatedSubscribeAndSwitchToStandbyClusterpassed 10/10 locally with a temporaryinvocationCount = 10(annotation not included in this PR).ReplicatedSubscriptionTestandReplicatedSubscriptionWithTransactionTestclasses pass locally.Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes