Skip to content

[Bug] A cleared group forbidden state is persisted as the literal "null" and then fails the config load on the next broker start #11429

Open

Description

@iloveeyjafjalla

Before Creating the Bug Report

  • I found a bug, not just asking a question, which should be created in GitHub Discussions.
  • I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.
  • I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

OS: any (broker with the RocksDB config manager: enableRocksDBStore=true, i.e. RocksDBSubscriptionGroupManager)

RocketMQ version

branch: develop
Git commit id: a7edeec

JDK Version

Compiler: Temurin 1.8.0_504 (target 1.8, same as CI)

Describe the Bug

The RocksDB variant of the subscription group manager persists the group's forbidden bits with JSON.toJSONString(this.getForbiddenTable().get(group)), and after the last bit of a group is cleared the table no longer holds an entry for that group — so the value written to the forbidden column family is the string "null":

@Override
public void updateForbidden(String group, String topic, int forbiddenIndex, boolean setOrClear) {
    super.updateForbidden(group, topic, forbiddenIndex, setOrClear);
    this.rocksDBConfigManager.put(FORBIDDEN_COLUMN_FAMILY_NAME, group,
        JSON.toJSONString(this.getForbiddenTable().get(group)));   // -> "null" once the group has no forbidden bits left
}

The same shape appears in setForbidden and clearForbidden (broker/src/main/java/org/apache/rocketmq/broker/config/v1/RocksDBSubscriptionGroupManager.java:287-306; the updateForbidden variant is at :276-284).

On the next start the stored value is fed back into the loader, which expects a JSON object of topic → bits:

protected void decodeForbidden(byte[] key, byte[] body) {
    JSONObject jsonObject = JSON.parseObject(new String(body, RocksDBConfigManager.CHARSET));
    Set<Map.Entry<String, Object>> entries = jsonObject.entrySet();   // NPE for a "null" body
    ...
}

(:263-273.) JSON.parseObject("null") returns null, so the NullPointerException leaves loadForbidden through its catch and the method returns false (:101-109), loadSubscriptionGroupAndForbidden() returns false (:95-99), then RocksDBSubscriptionGroupManager.load() returns false (:81-83), BrokerController.initializeMetadata() returns false (BrokerController.java:863) and BrokerStartup answers that with System.exit(-3) (BrokerStartup.java:287-291) — the broker does not start until that key is repaired by hand.

The trigger needs nothing more than un-forbidding the last forbidden topic of a group; with a single forbidden topic that is the very first clear, so no multi-topic scenario is required. (The related cross-topic loss when a group has several forbidden topics is reported separately in #11093 — this report is about the value that gets persisted and what it breaks on the next load; the two fixes compose.)

Steps to Reproduce

  1. Start a broker with enableRocksDBStore=true and forbid a topic for a group, e.g. via UPDATE_AND_GET_GROUP_FORBIDDEN (AdminBrokerProcessor.updateAndGetGroupForbidden, :466-489) with readable=false for group=g, topic=t.
  2. Un-forbid the same topic (readable=true). The forbidden column family of group now holds the literal null.
  3. Restart the broker: the config load fails and the process exits (initialize() false → System.exit(-3)), with loadForbidden exception … java.lang.NullPointerException at decodeForbidden.

At the unit level, RocksDBSubscriptionGroupManagerTest already injects a mocked RocksDBConfigManager (no real RocksDB needed): forbidding then clearing one topic shows the "null" value being written today, and calling decodeForbidden(key, "null".getBytes()) throws.

What Did You Expect to See?

Clearing a group's forbidden state persists "no forbidden state": the key is removed (or at least stores a valid empty JSON object), and the loader tolerates an already written null body so that brokers which ran with the broken value can still start after upgrading.

What Did You See Instead?

A "null" value is written, and the next broker start aborts because the loader dereferences the parsed null; the failure mode is a broker that cannot start from its own persisted configuration.

Additional Context

Minimal fix in RocksDBSubscriptionGroupManager: route the three overrides through one helper that deletes the column-family key when the group has no forbidden bits left instead of writing JSON.toJSONString(null), and ignore an empty (null) body in decodeForbidden so existing stores recover. The sibling JSON mode persists the whole table, which is why it never hits this.

I will open a PR with those changes plus regression tests in RocksDBSubscriptionGroupManagerTest (red before: a "null" value is written / decodeForbidden throws; green after: the key is deleted and a null body is skipped).

Activity

  1. RockteMQ-AI commented on Oct 14, 2026

    @RockteMQ-AI
    Contributor

    Issue Evaluation

    Category: type/bug | Status: Confirmed

    The reported issue has been verified against the current codebase.

    Root Cause: RocksDBSubscriptionGroupManager persists forbidden state via JSON.toJSONString(this.getForbiddenTable().get(group)) in updateForbidden, setForbidden, and clearForbidden (lines 276-306). When the last forbidden bit is cleared, getForbiddenTable().get(group) returns null, so the stored value becomes the literal string "null". On next startup, decodeForbidden calls JSON.parseObject("null") which returns null, causing NullPointerException at jsonObject.entrySet(). This propagates up and causes BrokerStartup to call System.exit(-3) — the broker refuses to start.

    Impact: Critical — clearing the last forbidden topic for a group renders the broker unstartable until the corrupted RocksDB key is manually repaired.

    Severity: Critical — broker startup failure, requires manual intervention.

    Proposed Fix: Guard against null before serializing — either skip the put when the forbidden table entry is null (and optionally delete the key), or serialize an empty JSON object {} instead of "null".

    The analysis is thorough and the reproduction path is clear. Looking forward to the PR.


    Automated evaluation by RockteMQ-AI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions