[fix][ml] Preserve ledger entries/size when transformLedgerInfo callback completes after a concurrent close

[dao-jun]

Motivation

tryTransformLedgerInfo (shared by per-ledger property add/remove and offload metadata updates) writes its result back to the in-memory ledgers map via a blind ledgers.put(ledgerId, newInfo), where newInfo carries entries/size/timestamp snapshotted at transform time. If the ledger fills and ledgerClosed updates those
fields during the async metadata-store round-trip, the callback overwrites them with the stale snapshot values. The regressed value is also captured by the next persisted metadata snapshot, so the wrong counts can survive a restart.

Modifications

1. Replace the blind put with a merge: keep the transform's changed fields (properties, offload context, ledger id) from newInfo, take entries/size/timestamp from the current in-memory value.
2. Add tests

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

[lhotari]

Combined AI-assisted review (Claude + Codex), curated and verified by me.

The race is real and the fix is correct for its stated purpose. ledgerClosed holds only the ManagedLedger monitor — not metadataMutex — so it can update entries/size/timestamp in the in-memory ledgers map while a transform's metadata-store round-trip is in flight. The previous blind put overwrote those fields with the stale snapshot, and the next persisted snapshot made the regression durable. I verified that all other writers to those fields either hold metadataMutex (trimming, other transforms) or run during recovery before the ML is open, so ledgerClosed is exactly the racing writer this merge targets. Offload transforms are unaffected: they modify the nested OffloadContext (including its own timestamp), which copyFrom(incoming) preserves.

I've left inline comments with two suggestions to harden the merge (lightproto field-presence handling and the absent-key insert) and one on test coverage. Additional notes that don't map to diff lines:

1. Pre-existing, out of scope for this PR — the mirror direction of this race remains, and it's deterministic: ledgerClosed (ManagedLedgerImpl.java L1972-L1974) rebuilds the LedgerInfo from scratch (ledgerId/entries/size/timestamp only), dropping any PIP-404 per-ledger properties already present in the in-memory map. A property added to the open current ledger is silently lost when that ledger later closes normally — no race required. The new test only passes because in its ordering (close before callback) the merge re-applies the property on top of the rebuilt info; with the opposite (normal) ordering the property is gone. Worth a follow-up issue.

2. Backports: branch-4.0 still uses the protobuf-generated MLDataFormats (LedgerInfo.newBuilder()), so new LedgerInfo().copyFrom(...) won't compile there — the release/4.0.13 cherry-pick needs a toBuilder()-based equivalent. branch-4.2 already uses the lightproto style, so that one should apply cleanly.

3. Nit: the "Verifying this change" section of the description still shows all three template placeholders — the "added tests" option applies here.

[dao-jun]

1. Pre-existing, out of scope for this PR — the mirror direction of this race remains, and it's deterministic: ledgerClosed (ManagedLedgerImpl.java L1972-L1974) rebuilds the LedgerInfo from scratch (ledgerId/entries/size/timestamp only), dropping any PIP-404 per-ledger properties already present in the in-memory map. A property added to the open current ledger is silently lost when that ledger later closes normally — no race required. The new test only passes because in its ordering (close before callback) the merge re-applies the property on top of the rebuilt info; with the opposite (normal) ordering the property is gone. Worth a follow-up issue.

fixed by #26227