[swuferhong]

Purpose

Linked issue: close #3613

Add tabletServer resource reporting and a coordinator-side KvLeaderReplicaCapacityManager to protect clusters from creating too many KV leader replicas.

TabletServers now report optional CPU and memory information during registration. The coordinator derives kvLeaderReplicaCapacity from live tabletServer memory and kv.leader-replica.memory-reserved, tracks the current KV leader replica count in memory, rebuilds it on coordinator leadership startup, and checks/reserves capacity when creating KV tables or partitions.

The capacity limit is disabled unless more than half of live tablet servers report memory. Unknown-memory TabletServersuse the average known memory when the limit is enabled.

Also add dynamic config support, metrics, RPC error mapping, serde compatibility, auto-partition accounting, and tests for resource probing, capacity calculation, manual create/drop, and auto partition create/drop.

Brief change log

Tests

API and Format

Documentation

[Copilot]

Pull request overview

This PR adds a coordinator-side capacity guard for KV leader replicas, based on live TabletServer-reported resources (primarily memory), to prevent creating excessive KV leader replicas and reduce OOM risk in TabletServers.

Changes:

• Add TabletServer CPU/memory probing + registration reporting, with JSON serde version bump and backward compatibility handling.
• Introduce KvLeaderReplicaCapacityManager in the coordinator, integrate it into create/drop table & partition flows (including auto-partition), and expose metrics.
• Add dynamic config support, RPC error mapping, and unit/integration tests for probing and capacity accounting.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
fluss-server/src/test/java/org/apache/fluss/server/zk/data/TabletServerRegistrationJsonSerdeTest.java	Updates serde tests for new registration version and optional resource fields.
fluss-server/src/test/java/org/apache/fluss/server/tablet/TabletServerResourceProbeTest.java	Adds tests for resource probing from config/cgroup/OS fallbacks.
fluss-server/src/test/java/org/apache/fluss/server/DynamicConfigChangeTest.java	Adds test for dynamic update of KV leader replica reserved memory config.
fluss-server/src/test/java/org/apache/fluss/server/coordinator/TableManagerITCase.java	Adds IT coverage for capacity enforcement on table and partition operations.
fluss-server/src/test/java/org/apache/fluss/server/coordinator/KvLeaderReplicaCapacityManagerTest.java	Adds unit tests for capacity calculation, accounting, and rebuild behavior.
fluss-server/src/test/java/org/apache/fluss/server/coordinator/AutoPartitionManagerTest.java	Ensures auto-partition create/drop reserves/releases capacity.
fluss-server/src/main/java/org/apache/fluss/server/zk/data/TabletServerRegistrationJsonSerde.java	Bumps registration JSON version and serializes/deserializes CPU/memory fields.
fluss-server/src/main/java/org/apache/fluss/server/zk/data/TabletServerRegistration.java	Extends registration model to carry TabletServerResource.
fluss-server/src/main/java/org/apache/fluss/server/tablet/TabletServerResourceProbe.java	Implements probing of CPU/memory from config, cgroups, or OS/JVM.
fluss-server/src/main/java/org/apache/fluss/server/tablet/TabletServer.java	Includes probed resources in tablet server registration.
fluss-server/src/main/java/org/apache/fluss/server/metadata/TabletServerResource.java	Adds a resource value object (cpu/memory) for reporting/aggregation.
fluss-server/src/main/java/org/apache/fluss/server/metadata/ServerInfo.java	Extends server metadata to include TabletServerResource.
fluss-server/src/main/java/org/apache/fluss/server/metadata/CoordinatorMetadataCache.java	Exposes “live” tablet servers (excluding permanently-offline) for capacity calc.
fluss-server/src/main/java/org/apache/fluss/server/DynamicServerConfig.java	Allows kv.leader-replica.memory-reserved to be dynamically reconfigured.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/KvLeaderReplicaCapacityManager.java	New coordinator component calculating capacity and tracking reserved KV leader replicas.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/event/watcher/TabletServerChangeWatcher.java	Propagates tablet server resource info into coordinator metadata updates.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorService.java	Enforces/reserves/releases capacity during create/drop table/partition operations.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorServer.java	Wires manager into coordinator lifecycle, registers metrics + dynamic config, rebuilds counts on leadership.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorEventProcessor.java	Includes resource info when building initial server metadata in coordinator context.
fluss-server/src/main/java/org/apache/fluss/server/coordinator/AutoPartitionManager.java	Reserves/releases capacity during auto partition create/drop paths.
fluss-rpc/src/main/java/org/apache/fluss/rpc/protocol/Errors.java	Adds RPC error mapping for insufficient KV leader replica capacity.
fluss-common/src/main/java/org/apache/fluss/metrics/MetricNames.java	Adds metric names for KV leader replica count/capacity gauges.
fluss-common/src/main/java/org/apache/fluss/exception/InsufficientKvLeaderReplicaCapacityException.java	Introduces a public API exception for capacity exhaustion.
fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java	Adds new config options for reserved memory per KV leader replica and tablet-server resource reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wuchong]

Thanks @swuferhong , I commented the major concern here:

Avoid maintaining an exactly-once KV leader count outside CoordinatorContext

currentKvLeaderReplicaCount creates a second mutable representation of coordinator state. Keeping it accurate requires reservation, rollback and release logic across every table/partition CRUD path, while failover rebuild performs an expensive and non-atomic metadata scan.

Please maintain an event-thread-owned Set<TableBucket> in CoordinatorContext instead. Add observed KV buckets on create, conservatively treat unknown orphan assignments as KV during failover, and remove them idempotently from onDeleteTableBucket as ResumeDropEvents are admitted by the lifecycle throttler. KvLeaderReplicaCapacityManager should only keep a volatile snapshot derived from kvBuckets.size() and perform a best-effort observed + requested > capacity check, without reservation, rollback, decrease, or full rebuild logic.

[wuchong]

Btw, Could we rename KvLeaderReplicaCapacityManager to ReplicaCapacityController?

Replica provides a better component boundary because capacity control may later cover log replicas and KV hot replicas in addition to KV leader replicas. Controller is also more accurate than Manager: this component does not manage replica lifecycles; it makes creation admission decisions based on observed replica counts and cluster resource capacity.

However, the current methods, metrics, and configuration should remain explicitly scoped to KV leaders, since that is the only replica type currently supported.

[swuferhong]

Hi, @wuchong comments addressed. PTAL, thanks.

[wuchong]

Is the failed case related to this change?

By the way, I believe it is acceptable if the auto-partition manager starts before the metadata cache update, as this does not cause server failure. The task will be rescheduled at the next interval, and the partition will eventually be created. On the other hand, we should add a delay (e.g., 1min) to the schedule in the auto-partition manager instead of using 0. Starting immediately is risky because the server is typically busy during startup.

[swuferhong]

This should have been caused by the state before commit 004e514, where metadataCache was initialized twice, so the metadataCache instance passed as a constructor argument to ReplicaCapacityController got overwritten.

Adding the delay may cause many ITCases that rely on auto partition to fail. We'd need to expose the delay as a config parameter so it can be configured in tests. Could we do this in a follow-up PR?

[swuferhong]

Hi, @wuchong comments addressed and ci passed.

[wuchong]

LGTM.