Skip to content

[fix][test] Fix flaky ExtensibleLoadManagerImplTest by re-serving the channel topic in initializeState - #25976

Merged
lhotari merged 1 commit into
apache:masterfrom
merlimat:mmerli/fix-flaky-extensible-loadmanager-channel-unserved
Jun 8, 2026
Merged

[fix][test] Fix flaky ExtensibleLoadManagerImplTest by re-serving the channel topic in initializeState#25976
lhotari merged 1 commit into
apache:masterfrom
merlimat:mmerli/fix-flaky-extensible-loadmanager-channel-unserved

Conversation

@merlimat

@merlimat merlimat commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Motivation

The Pulsar CI Flaky suite has been failing frequently on ExtensibleLoadManagerImplTest, with a 2-minute timeout in the initializeState @BeforeMethod (example run: https://github.com/apache/pulsar/actions/runs/27144684649):

org.awaitility.core.ConditionTimeoutException: Assertion condition null within 2 minutes.
    at ...ExtensibleLoadManagerImplBaseTest.initializeState(ExtensibleLoadManagerImplBaseTest.java)

Root cause. Tests such as testHandleNoChannelOwner deliberately churn leader election by closing the LeaderElectionService on both brokers. This can leave the channel-topic bundle pulsar/system/0x00000000_0xffffffff (which hosts loadbalancer-service-unit-state) in an owner-recorded-but-not-actually-served state. Every channel operation then fails with ... not served by this instance ... Please redo the lookup.

initializeState (reworked in #25946) drives monitor() and retries the namespace unload for 120s, but monitor() cannot heal this particular state: ExtensibleLoadManagerImpl.handleNoChannelOwnerError only restarts leader election when the channel reports "no channel owner now". When an owner is recorded but refuses to serve, no such error is thrown, recovery never triggers, and the unload — which must publish to the channel topic — can never succeed. The 120s budget is exhausted and the @BeforeMethod fails, cascading to skipped tests.

Modifications

In ExtensibleLoadManagerImplBaseTest.initializeState, force-serve the channel topic inside the existing retry loop, before the unload:

  • admin.lookups().lookupTopic(...) re-assigns the pulsar/system bundle, and
  • admin.topics().getStats(...) forces the recorded owner to actually load it (the lookup layer alone can claim an owner that refuses to serve).

This is the same sequence awaitChannelOwnerStable() already uses to stabilize after churn, but run on every retry attempt so the channel is re-served immediately before each unload — rather than only once in the churn test's finally, where the state can degrade again before the next initializeState. It is guarded to the ServiceUnitStateTableViewImpl (system-topic) variant, matching awaitChannelOwnerStable's own guard; the metadata-store variant has no channel topic to serve.

This is a test-side mitigation. The durable fix is product-side — teaching monitor() / handleNoChannelOwnerError to detect owner-recorded-but-unserved and re-assign the bundle — and can follow separately.

… channel topic in initializeState

The initializeState @BeforeMethod could time out after 2 minutes when a prior
test (e.g. testHandleNoChannelOwner) churned leader election and left the
channel-topic bundle (pulsar/system, loadbalancer-service-unit-state) in an
owner-recorded-but-unserved state. monitor() does not heal that state
(handleNoChannelOwnerError only restarts election on the "no channel owner now"
error), so the namespace unload could never publish to the channel.

Force-serve the channel topic on every retry attempt via an admin lookup +
getStats (the same sequence awaitChannelOwnerStable already uses), so the unload
can proceed. Guarded to the system-topic table-view variant.
@lhotari

lhotari commented Jun 8, 2026

Copy link
Copy Markdown
Member

It seems that this didn't fully fix the problem since there was a failure: https://github.com/apache/pulsar/actions/runs/27153833000/job/80153890688?pr=25976#step:8:2151
I presume that #25977 is also needed.

@lhotari
lhotari merged commit b50a303 into apache:master Jun 8, 2026
42 of 43 checks passed
@lhotari

lhotari commented Jun 8, 2026

Copy link
Copy Markdown
Member

#25977 has been modified to take the changes into account from this PR. Please review #25977

@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 10, 2026
lhotari pushed a commit that referenced this pull request Jun 10, 2026
lhotari pushed a commit that referenced this pull request Jun 10, 2026
priyanshu-ctds pushed a commit to datastax/pulsar that referenced this pull request Jun 15, 2026
… channel topic in initializeState (apache#25976)

(cherry picked from commit 136bc88)
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants