Skip to content

[fix][broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl - #26031

Merged
lhotari merged 1 commit into
apache:masterfrom
BewareMyPower:bewaremypower/fail-fast-load-balancer-incorrect-config
Jun 15, 2026
Merged

[fix][broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl#26031
lhotari merged 1 commit into
apache:masterfrom
BewareMyPower:bewaremypower/fail-fast-load-balancer-incorrect-config

Conversation

@BewareMyPower

Copy link
Copy Markdown
Contributor

Motivation

We observed production issues due to the following misconfigurations:

loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.AvgShedder

Logs:

2026-06-15T07:27:18,079+0000 [main] WARN  org.apache.pulsar.broker.loadbalance.LoadManager - Error when trying to create load manager:
  java.lang.IllegalArgumentException: The load shedding strategy: org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder can't work with the placement strategy: org.apache.pulsar.broker.loadbalance.impl.AvgShedder

This did not fail fast, instead, it created a SimpleLoadManagerImpl object as a load balance. This early implementation has many issues and its functionality is hardly tested, for example, it might not work well with multiple listeners specified like the broker do not have gw listener.

IMO, we should remove this outdated implementation in future, but for now, I think we should fail fast for misconfigurations rather than falling back to this implementation.

@BewareMyPower BewareMyPower self-assigned this Jun 15, 2026
@BewareMyPower BewareMyPower added release/4.2.3 release/4.0.12 type/bug The PR fixed a bug or issue reported a bug area/broker labels Jun 15, 2026
@BewareMyPower BewareMyPower changed the title [fix][broker] Don't fall back to SimpleLoadManagerImpl for misconfigurations [fix][broker] Fail fast for load balancer misconfigurations instead of falling back to SimpleLoadManagerImpl Jun 15, 2026
@nodece

nodece commented Jun 15, 2026

Copy link
Copy Markdown
Member

@BewareMyPower This fix should only be released in 5.0. Otherwise, I'm worried that Pulsar may fail to work when users provide an incorrect configuration.

@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 15, 2026

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lhotari
lhotari merged commit c3f8c05 into apache:master Jun 15, 2026
45 checks passed
@BewareMyPower
BewareMyPower deleted the bewaremypower/fail-fast-load-balancer-incorrect-config branch June 15, 2026 13:01
@BewareMyPower

Copy link
Copy Markdown
Contributor Author

@nodece Falling back to SimpleLoadManagerImpl is much worse. Could you imagine a cluster running correctly with SimpleLoadManagerImpl configured? It would even make diagnosis much harder.

@lhotari

lhotari commented Jun 15, 2026

Copy link
Copy Markdown
Member

@BewareMyPower This fix should only be released in 5.0. Otherwise, I'm worried that Pulsar may fail to work when users provide an incorrect configuration.

@nodece I think that with k8s clusters, with proper poddistruptionbudget config (or similar), the broker sts rollout would stop if a broker doesn't come up after upgrading. The user could dig into logs if that happens and prevents an upgrade.

lhotari pushed a commit that referenced this pull request Jun 22, 2026
…f falling back to SimpleLoadManagerImpl (#26031)

(cherry picked from commit c3f8c05)
lhotari pushed a commit that referenced this pull request Jun 22, 2026
…f falling back to SimpleLoadManagerImpl (#26031)

(cherry picked from commit c3f8c05)
sandeep-ctds pushed a commit to datastax/pulsar that referenced this pull request Jul 31, 2026
…f falling back to SimpleLoadManagerImpl (apache#26031)

(cherry picked from commit c3f8c05)
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Aug 28, 2026
…f falling back to SimpleLoadManagerImpl (apache#26031)

(cherry picked from commit c3f8c05)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants