docs(ENG-WEIGHT-RESIDENCY): delete the staging-fraction knob and the comment that outlived its reader - #2438

localai-org-maint-bot · 2026-08-31T22:34:31Z

docs/ENVIRONMENT.md documented VT_QWEN35_STAGE_MIN_FREE_FRAC in the user-facing table — a default of 0.55, an exact formula, tuning direction — and nothing read it. Its only occurrence in compiled code was a // comment.

Closes #2385.

Why it was dead

The staging policy was rewritten to decide ONCE from a stable total (#2342). SetSafetensorsWeightBudget reads only VT_QWEN35_STAGE_RESERVE_BYTES and VT_QWEN35_ALIAS_HOST_WEIGHTS and applies 2*model + reserve <= total — a rule with no fraction in it. The knob's reader was deleted with the old live-free policy; the doc row and a 23-line header comment survived it.

Worse than silence

The row told the reader that a bad value degrades gracefully — "a typo cannot silently disable the floor". In fact every value is discarded, including a valid one. An operator tuning a staging OOM would change it, see no effect, and have nothing to explain why.

The header comment was worse than the issue described: 23 lines describing the deleted per-weight live-free rule, run together with no blank line into the current StagingFitsModel doc comment, so the two read as one block with the false half first. It now states the rule in full — 2 * model_weight_bytes + reserve_bytes <= device_total_bytes, latched once per process — and says explicitly that nothing is asked per weight and no fraction of live free memory is read.

The row had already been deleted once, and a merge resurrected it

c31b2496e — subject literally "and drop the knob it replaced" — removed it on 2026-08-30. A branch cut in parallel against the same red check-env-doc added the RESERVE_BYTES row beside the fraction rather than in place of it, describing the two as live siblings. The merge took both sides and the deliberately deleted row came back.

That is the automerge-applies-both hazard AGENTS.md names for shared record surfaces, and it is recorded here and in the matrix row so a third return is visible rather than mysterious.

Worth noting check-env-doc could not have caught this even in the reverse-grep form being added under #2389: its scan matches quoted string literals, and this knob's sole code occurrence was a backticked mention inside a comment — exactly the case that issue flags as the one a naive reverse check would miss.

Scope

Documentation plus one stale code comment. Zero compiled changes — the header diff is comment lines only, verified by filtering the diff for non-comment edits.

The owning matrix row stays INVENTORIED rather than moving to DONE: tests/scripts/test_agent_record.py asserts that state, and a DONE row would owe a spec with an ## Outcome, which is outside a doc-and-comment scope. The row now says why — nothing yet prevents the recurrence.

Gates

check-env-doc rc=0 (398 documented), check-agent-record ENGINE count unchanged, test_agent_record 120 OK, test_check_env_doc 8 OK, commit style and trailers OK.

Coordination

#2389 adds the reverse-direction check and currently allowlists this variable with a self-clearing staleness guard. Landing this PR makes that entry stale by design, and its checker then says so by name — verified locally:

ERROR: UNREAD_EXCEPTIONS entr(ies) ... are stale -- the variable has left the
user-facing table or gained a reader ... Delete the entry.

So this PR must land BEFORE #2389, and #2389 drops the entry. That ordering is deliberate: the debt cannot outlive its justification.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

…omment that outlived its reader `docs/ENVIRONMENT.md` carried `VT_QWEN35_STAGE_MIN_FREE_FRAC` in the USER-FACING table with a default of `0.55`, an exact formula, a parse grammar and a tuning direction, and nothing in the tree read it. Its only occurrence in compiled code was a `//` comment in `include/vllm/model_executor/models/qwen3_5_weights.h`, which is why `scripts/check-env-doc.py` never saw it: that checker matches a QUOTED string literal precisely so a bare mention in prose is not counted as a read, and it only looks for scanned-but-undocumented, never the reverse. The reader was deleted with the policy it belonged to. #2342 rewrote staging to decide ONCE from two stable numbers, and `SetSafetensorsWeightBudget` (`src/vllm/model_executor/models/qwen3_5_weights.cpp:211`) now reads exactly two variables, `VT_QWEN35_STAGE_RESERVE_BYTES` and `VT_QWEN35_ALIAS_HOST_WEIGHTS`, to apply `2 * model_weight_bytes + reserve <= device_total_bytes`. There is no fraction anywhere in that rule. The row and the header comment survived the function they described. THE ROW WAS WORSE THAN SILENCE, and its own reassurance is the reason. It told the reader that an unset, empty, unparsable, `<= 0` or `>= 1` value "falls back to 0.55 rather than refusing, so a typo cannot silently disable the floor". The truth is stronger and points the other way: EVERY value is discarded, a correctly typed one included. A reader who set `0.75` to be conservative on a shared box got the same behaviour as one who set nothing, and the sentence designed to reassure them is what guaranteed they would not check. THE ROW HAD ALREADY BEEN DELETED ONCE, AND A MERGE PUT IT BACK. `c31b2496e` (#2359), whose subject is literally "and drop the knob it replaced", removed it on 2026-08-30. `a81d6e52a` was cut in parallel against the same red gate, added the `RESERVE_BYTES` row BESIDE the fraction rather than in place of it, and its body described the pair as live siblings where the fraction "asks the box about LIVE free memory per weight". Merge `ecc81e64c` took both sides, and a deliberately deleted row came back to `main` carrying a description of a policy that no longer existed. Deleting it a second time without recording that is how it would return a third. The header comment is the second half, and it was the worse half. Twenty-three lines describing the DELETED per-weight live-free rule sat directly above `StagingFitsModel`, which implements the new one, with no blank line between them, so the two read as a single block and the stale half came first. A comment that contradicts the function beneath it is worse than no comment, because it is the only description a reader has and it is confident. It now states the live rule -- the stable-total arithmetic, the reserve variable that feeds it, the latch that runs once per process -- and says plainly that no fraction of live free memory is read, pointing at `SetSafetensorsWeightBudget`'s own comment for what asking per weight cost when it was measured. No compiled code changes. The header diff is comment lines only, and the rule, its tests and its behaviour are untouched. The `ENG-WEIGHT-RESIDENCY` row is reconciled in the same change, since this change is what made it stale. It stays `INVENTORIED` rather than `DONE`: nothing yet PREVENTS the recurrence, and that gate is `ENG-GATE-ENV-DOC` under #2389, which owns the reverse-direction check and its tests. `check-env-doc.py` reports OK over 398 names, `check-agent-record.py` reports ENGINE=178 unchanged, and `tests/scripts/test_agent_record.py` is green at 120 tests, including the case that pins this row's ID, matrix and state. FOLLOWING_AGENTS_PROTOCOL Closes #2385 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]

mudler merged commit 8c1d8d5 into main Aug 31, 2026
2 of 17 checks passed

Navigation Menu

docs(ENG-WEIGHT-RESIDENCY): delete the staging-fraction knob and the comment that outlived its reader - #2438