Skip to content

[fix][meta] Run ledger-underreplication notification callbacks off the metadata-store listener thread - #26065

Merged
lhotari merged 1 commit into
apache:masterfrom
merlimat:mmerli/fix-underreplication-notification-blocking
Jun 22, 2026
Merged

[fix][meta] Run ledger-underreplication notification callbacks off the metadata-store listener thread#26065
lhotari merged 1 commit into
apache:masterfrom
merlimat:mmerli/fix-underreplication-notification-blocking

Conversation

@merlimat

Copy link
Copy Markdown
Contributor

Motivation

PulsarLedgerUnderreplicationManager.handleNotification is registered as the metadata-store change listener and runs on the metadata-store notification thread, inside synchronized(this). When a relevant z-node changed, it invoked the registered BookKeeper GenericCallbacks (replicationEnabledCallbacks / lostBookieRecoveryDelayCallbacks) inline. Those callbacks can perform synchronous metadata-store reads (the class itself documents this near notifyUnderReplicationLedgerChanged), so they blocked the metadata-store notification thread — and held the manager's monitor — while doing metadata I/O.

Modifications

Run the registered callbacks on a dedicated single-threaded executor instead of the notification thread. The callback list is still snapshotted and cleared synchronously under the monitor, and notifyAll() (the primary wait/notify used by the blocking poll methods) stays synchronous; only the callback invocation is offloaded, so it runs off the notification thread and without holding the monitor. The single thread preserves notification ordering, and close() shuts the executor down. Callbacks re-read current state when they re-register, so this is eventually consistent.

Verifying this change

Covered by existing LedgerUnderreplicationManagerTest (87 executions across metadata-store implementations), which awaits callbacks via Awaitility/latches — passes.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

…e metadata-store listener thread

PulsarLedgerUnderreplicationManager.handleNotification is registered as the
metadata-store change listener and runs on the metadata-store notification
thread, inside synchronized(this). When a relevant z-node changed, it invoked the
registered BookKeeper GenericCallbacks (replicationEnabledCallbacks /
lostBookieRecoveryDelayCallbacks) inline. Those callbacks can perform synchronous
metadata-store reads (the class documents this near
notifyUnderReplicationLedgerChanged), so they blocked the metadata-store
notification thread -- and held the manager's monitor -- while doing metadata IO.

Run the registered callbacks on a dedicated single-threaded executor instead of
the notification thread. The callback list is still snapshotted and cleared
synchronously under the monitor, and notifyAll() (the primary wait/notify used by
the blocking poll methods) stays synchronous; only the callback invocation is
offloaded, so it runs off the notification thread and without holding the
monitor. The single thread preserves notification ordering, and close() shuts the
executor down. Callbacks re-read current state when they re-register, so this is
eventually consistent.
@lhotari
lhotari merged commit cc96d1c into apache:master Jun 22, 2026
44 checks passed
@lhotari lhotari added this to the 5.0.0-M2 milestone Jun 22, 2026
lhotari pushed a commit that referenced this pull request Jun 22, 2026
…e metadata-store listener thread (#26065)

(cherry picked from commit cc96d1c)
lhotari pushed a commit that referenced this pull request Jun 22, 2026
…e metadata-store listener thread (#26065)

(cherry picked from commit cc96d1c)
sandeep-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 31, 2026
…e metadata-store listener thread (apache#26065)

(cherry picked from commit cc96d1c)
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Aug 28, 2026
…e metadata-store listener thread (apache#26065)

(cherry picked from commit cc96d1c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants