spec(tenstorrent): residual-RMS golden probe at the rows>=32 device boundary - #393

lu-zero · 2026-08-11T16:15:46Z

What

Owed evidence for the bot-flagged numerics divergence at the residual-RMS device boundary in tenstorrent_ops.cpp::RmsNormKernel (BACKEND-TENSTORRENT-RESIDUAL-GOLDEN, child of BACKEND-TENSTORRENT).

At rows >= kDeviceResidualMinRows == 32 the device path does ttnn::add + ttnn::rms_norm in bf16; the host/CPU path (cpu_ops.cpp:371) accumulates the variance in f32. No op-level cross-device golden existed at the boundary — the only TT numerics evidence was the coarse e2e greedy golden. This closes that gap.

Measured (real Blackhole P150, `blackhole-94712C24111071E4`)

RED-first op-level probe through the public vt::RmsNorm/vt::Backend seam. D=1024 (Qwen3-0.6B hidden width), f32 residual, deterministic LCG inputs:

rows	path	max_abs vs CPU f32	max_rel
1, 31	host (f32)	0	0
32, 33	device bf16	0.0459	1.94
64	device bf16	0.0459	1.95
128	device bf16	0.0459	2.57

22/22 cases, 826/826 assertions, clean exit. The bot's flag was exactly right: below rows=32 the host path is bit-identical to CPU; at/above it the device bf16 path diverges. The constant max_abs across all device row-counts is the signature of bf16 rounding on a single near-zero element, not accumulation with rows; the large max_rel is that same small absolute delta divided by a near-zero CPU reference.

Verdict: accept

vLLM's own bf16 RmsNorm carries the same rounding (the CPU f32 oracle is stricter than vLLM's runtime).
The rows>=32 threshold is load-bearing for perf (the residency handoff recorded that "always device residual" regressed Qwen3-0.6B ~12.3→10.5 tok/s; the hybrid threshold restored it).
No src/ change — this PR adds a test + spec + records only. The residual path under test is the already-landed code that produced our_ids_tenstorrent.npy, so the committed e2e anchor is by construction unchanged.

What's NOT claimed

The e2e gate (test_qwen3_paged_engine vs our_ids_tenstorrent.npy) was attempted but did not complete: the paged-attention shapes JIT-compile cold in ~30 min on this P150. A vllm-cli smoke confirmed 12.5 tok/s warm + run-to-run deterministic greedy. The full paged-gate reproduction is recorded as a resume recipe in the spec for a long uninterrupted window — it is a reproduction of an already-recorded result, not a gate on a code change (this PR touches no src/).
This row's declared gate is the op-level numerics probe (passed). The e2e golden is the gate for the parent BACKEND-TENSTORRENT row, not this one.

Review

Fresh static + targeted-mutation review (read-only, no session context) returned PASS. Non-vacuity proven via M3 (tighten max_abs < 1e-5f → RED at rows>=32 device path, GREEN at rows<32 host path). Two MINOR findings addressed (spec input-range typo, defensive comment on the load-bearing residual arg).

Files

NEW .agents/specs/tenstorrent-residual-golden.md (spec + measured ## Outcome)
NEW TEST_CASE in tests/vt/test_tenstorrent_backend.cpp (public seam only, no ttnn headers in the test TU)
NEW BACKEND-TENSTORRENT-RESIDUAL-GOLDEN row in .agents/backend-matrix.md
NEW claim in .agents/coordination.md
scripts/check-agent-record.py BACKEND count 80→81
No src/ change.

…oundary Owed evidence for the bot-flagged numerics divergence at the residual-RMS device boundary in tenstorrent_ops.cpp::RmsNormKernel (BACKEND-TENSTORRENT -RESIDUAL-GOLDEN, child of BACKEND-TENSTORRENT). At rows >= kDeviceResidualMinRows == 32 the device path does ttnn::add + ttnn::rms_norm in bf16; the host/CPU path (cpu_ops.cpp:371) accumulates the variance in f32. No op-level cross-device golden existed at the boundary; this closes that gap. No src/ change — the residual path under test is the already-landed code that produced our_ids_tenstorrent.npy, so the committed e2e anchor is by construction unchanged. Measured on real Blackhole P150 (blackhole-94712C24111071E4), RED-first op-level probe through the public vt::RmsNorm/vt::Backend seam. D=1024 (Qwen3-0.6B hidden width), f32 residual, deterministic LCG inputs in [-1,1): rows=1,31 host path -> max_abs 0 (bit-identical to CPU) rows=32,33 device bf16 -> max_abs 0.0459 (rel 1.94x on near-zero outputs) rows=64 device bf16 -> max_abs 0.0459 (rel 1.95x) rows=128 device bf16 -> max_abs 0.0459 (rel 2.57x) 22/22 cases, 826/826 assertions, clean exit. The bot's flag on mudler#289 was exactly right: below rows=32 the TT host path is bit-identical to CPU; at rows>=32 the device bf16 path diverges. The constant max_abs across all device row-counts is the signature of bf16 rounding on a single near-zero output element, not accumulation with rows; the large max_rel is that same small absolute delta divided by a near-zero CPU reference (RMSNorm output crossing zero). Verdict: accept. vLLM's own bf16 RmsNorm carries the same rounding (the CPU f32 oracle is stricter than vLLM's runtime), and the rows>=32 threshold is load-bearing for perf (the residency handoff recorded that "always device residual" regressed Qwen3-0.6B ~12.3->10.5 tok/s; the hybrid threshold restored it). The e2e tie-break against our_ids_tenstorrent.npy blocked on cold-cache paged-JIT wall-time (~30 min on P150); a vllm-cli smoke confirmed 12.5 tok/s warm + run-to-run deterministic greedy, and a fresh static + targeted-mutation review returned PASS (non-vacuity via M3: tighten max_abs < 1e-5f trips RED at rows>=32 device path, GREEN at rows<32 host path). The full paged-gate reproduction is recorded as a resume recipe in the spec; it is a reproduction of an already-recorded result, not a gate on a code change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Maki:glm-5.2 [Maki]

… golden probe (mudler#393) Lands Luca Barbato's work from PR mudler#393, rebuilt on current main. Merged --no-ff and committed with --author so `git log` keeps `author=Luca Barbato`; a squash would have re-authored it to whoever pressed the button. Files BACKEND-TENSTORRENT-RESIDUAL-GOLDEN, the child row owing residual-RMS numerics where the bf16 device path takes over at rows>=32, against a CPU f32 oracle. Its CI red was five jobs, and only two were its own. agent-record, documentation-checkpoint's NOW demand and cuda-fat-build's Marlin gencode audit were all stale base -- the branch sat 92 commits behind, predating both mudler#374 and mudler#407. The two real findings were fixed rather than waived, and both had to be folded into the single commit because those gates are PER COMMIT: check-doc-checkpoint wanted docs/FEATURES.md for a feature-surface change. Right in substance: the row exists because those numerics are NOT yet evidenced, and a FEATURES.md reader had no way to know. The existing Tenstorrent line now names both outstanding items, the 16x16 rerun and the residual-RMS numerics. check-pr-size wanted evidence for the BACKEND count 80 -> 81 in the paired suite. A bump with no row behind it looks exactly like a bump for a new row, which is what the pin exists to catch, so two cases now tie the bump to this row. Also dropped from the branch tip: a deletion of §6i of dspark-spec-decode.md, 44 lines of measured text-parity evidence against the pinned oracle. Stale-base artifact -- that section landed on main after the branch last merged -- and landing it would have silently removed an evidence record. Main's version kept. Gate: scripts/agent-preflight.sh rc=0 on the row branch. CI is queue-blocked (mudler#274), so the operator's own gate run is the authority, as AGENTS.md prescribes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

@11k

…dK-WMMA (mudler#41) Lands bakon11's work from PR mudler#317, split out of mudler#228. Merged --no-ff so the contributor commits keep their authorship rather than being re-authored to whoever pressed the button. FP8 resident MoE experts and a SharedK-WMMA prefill path for Gemma-4 on RDNA4 ROCm, plus a CPU seam test that pins the portable vt::ExpertGeGLUFp8* symbols link and the recipe env knobs parse inertly. The concern this PR was originally held on is CLOSED BY MEASUREMENT, not by reading the diff. It was held because ~1,600 added lines sat in gemma4.cpp and gemma4_moe.cpp and Gemma-4 is a gate model. bakon11 rebuilt it to drop the CUDA forward restructure entirely, and both arms were then gated on dgx GB10 in one flock window: `gmain` (main) and `g317` (main + this PR) each returned gemma4-E4B STRICT 32/32 token-exact vs the vLLM 0.25.0 golden. gemma4.cpp is byte-identical to main. Its cuda-fat-build red was never its own: the gencode audit failed on seven files, every one of them Marlin, while this branch touches ROCm and gemma4_moe.cpp and no Marlin gencode at all. That was the drift mudler#407 fixed and the branch predated it -- the same stale-base class that also explained mudler#383 and mudler#393 tonight. ACCEPTED WITH A NAMED RESIDUAL, recorded because it is a product decision and not a gate result. This flips four knobs DEFAULT-ON for ROCm: VT_GEMMA4_PREFILL_PEER_ACT, VT_ATTN_PREFILL_FLASH_SHAREDK, VT_ATTN_PREFILL_SHAREDK_WMMA and VT_ROCM_GRAPH. Non-ROCm builds are unaffected -- every new lever is env-gated off or behind moe.enabled / ROCm ExpertGeGLU dispatch -- and each knob is individually reversible by env. But there is NO AMD CI runner, so those defaults rest entirely on the contributor's lab evidence (2x AMD Radeon AI PRO R9700 gfx1201 / RDNA4, ROCm 7.x, gemma-4-26B-A4B-it-fp8; prefill ~2k @11k, short decode ~45-52 t/s). Feature correctness is unprovable on GB10 and honestly so: all nine cached Gemma checkpoints there are dense bf16 with enable_moe_block false, so RunGemma4Moe, EnsureGemma4Fp8* and ExpertGeGLU* are simply unreachable. The GB10 gate proves NON-REGRESSION, never the feature. Gate: scripts/agent-preflight.sh rc=0 on the merged tree; check-agent-record, check-doc-checkpoint, check-public-doc-tables, check-now-current, check-env-doc and check-commit-trailers all green. CI is queue-blocked (mudler#274), so the operator's own gate run is the authority, as AGENTS.md prescribes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

lu-zero force-pushed the row/BACKEND-TENSTORRENT-RESIDUAL-GOLDEN branch from 4759544 to b00842e Compare August 11, 2026 16:18

Navigation Menu

spec(tenstorrent): residual-RMS golden probe at the rows>=32 device boundary - #393