[fix][test] Fix flaky testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader - #26049
Merged
merlimat merged 1 commit intoJun 17, 2026
Merged
Conversation
…AfterCreateReader SystemTopicBasedTopicPoliciesServiceTest.testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader was flaky (apache#25081). The test closes a __change_events reader and then expects a follow-up prepareInitPoliciesCacheAsync() that reuses the closed reader to fail. Two timing hazards made it non-deterministic: 1. A closed reader's hasMoreEventsAsync() (Reader.hasMessageAvailableAsync()) can still answer from cached state instead of failing with AlreadyClosedException. When it does, initPolicesCache() reaches the end of the topic and completes successfully, so the expected exception never happens. 2. After the close-triggered cleanup, a background topic load can re-run prepareInitPoliciesCacheAsync() for the namespace and leave a completed init future behind. The next call then short-circuits through the existing-future branch instead of re-initializing, and never throws. Make the test deterministic (test-only change): reuse the closed reader through a spy whose hasMoreEventsAsync() always fails with AlreadyClosedException; drop any lingering init future right before the follow-up call so it re-initializes; and stub createSystemTopicClient() so every reader created for the namespace is the failing spy, ensuring whichever initialization wins the race fails in the same asserted way. All existing assertions are unchanged. Verified by running the test 70 times (40 in-process + 30 with --rerun-tasks); it previously failed intermittently. Assisted-by: Claude Code (Claude Opus 4.8)
merlimat
approved these changes
Jun 17, 2026
sandeep-ctds
pushed a commit
to datastax/pulsar
that referenced
this pull request
Jul 31, 2026
…AfterCreateReader (apache#26049) (cherry picked from commit 6f82106)
nodece
pushed a commit
to ascentstream/pulsar
that referenced
this pull request
Aug 28, 2026
…AfterCreateReader (apache#26049) (cherry picked from commit 6f82106)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #25081
Motivation
SystemTopicBasedTopicPoliciesServiceTest.testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReaderis flaky. The test creates a real
__change_eventsreader, closes it, and then expects a follow-upprepareInitPoliciesCacheAsync(...)that reuses the closed reader to fail (so the cache-init cleanup pathruns). It fails intermittently in CI (e.g. an
AssertionErrorat theAssert.fail()afterprepareFuture.get(), or a timeout on a follow-up assertion).Two independent races, both rooted in the test relying on real asynchronous timing, cause the flakiness:
The closed reader does not reliably fail the read.
initPolicesCache(...)callsreader.hasMoreEventsAsync(), which delegates toReader.hasMessageAvailableAsync(). On a closedconsumer this is non-deterministic: it can still answer from cached state instead of failing with
AlreadyClosedException. When it does,initPolicesCache(...)reaches the end of the topic and completessuccessfully, so the expected exception never happens (and "Failed to check the move events for the system
topic" is never logged).
A background re-initialization leaves a completed init future behind. Closing the reader drives
readMorePoliciesAsync()intocleanPoliciesCacheInitMap(), which removes the namespace frompolicyCacheInitMap. But a background topic load can re-runprepareInitPoliciesCacheAsync(...)for thenamespace right after, leaving a completed init future. The follow-up
prepareInitPoliciesCacheAsync(...)then observes that future and short-circuits through the existing-future branch instead of
re-initializing — so it never throws.
This is a test-only fragility; the production code is correct and self-heals in both cases.
Modifications
Test-only change in
testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader, making the follow-upinitialization deterministically take the failure path the test asserts on:
hasMoreEventsAsync()deterministically fails withPulsarClientException.AlreadyClosedException, soinitPolicesCache(...)always fails at the read.policyCacheInitMap.remove(namespace)) right before the call so itre-initializes instead of short-circuiting on a previously completed future.
createSystemTopicClient(namespace)to return that failing spy, so whichever initialization wins therace (the test's, or a background re-init) fails in the same asserted way.
The assertions the test makes (exceptional completion;
policyCacheInitMap/readerCachescleaned; the"Failed to create reader" log absent; the "Failed to check the move events" log present;
cleanPoliciesCacheInitMap/cleanupFailedPolicyCacheIniteach invoked once; no "Recursive update") areunchanged — the fix only makes the conditions they assert on deterministic.
Verifying this change
This change is already covered by the existing test
SystemTopicBasedTopicPoliciesServiceTest.testPrepareInitPoliciesCacheAsyncThrowExceptionAfterCreateReader;the change makes it deterministic. Verified locally by running the test 70 times (40 in-process via
invocationCountand 30 in separate JVMs via--rerun-tasks), all green; before the fix the same loopsreproduced the failure.
Does this pull request potentially affect one of the following parts:
This is a test-only change and does not affect any of the following: dependencies, public API, schema,
default configuration values, threading model, binary protocol, REST endpoints, admin CLI options, metrics,
or deployment.