perf(gdn): optimize post-conv and causal-conv kernels - #155

richiejp · 2026-08-08T14:39:29Z

Row

KERNEL-SSM-MAMBA — Qwen GDN post-conv and causal-conv kernel sublevers; one row per PR.

Before starting

Issue/PR search and existing claim: this is a stacked follow-on to perf(gdn): dispatch exact causal-conv chunks #127, which supplies the exact causal-conv chunk descriptors measured here. No separate matching open PR was found for these two opt-in kernel specializations.
Roadmap or matrix row, plus scripts/ready-for-helper.py result when applicable: .agents/kernel-matrix.md:157 (KERNEL-SSM-MAMBA, INVENTORIED), feeding ROAD-V1-C2-LOCAL-BF16. Spikes: .agents/specs/sm120-qwen35-postconv-token-tile-2026-08-08.md and .agents/specs/sm120-qwen35-conv-channel-tile-2026-08-08.md.
Exact current-code and test/evidence anchors inspected: src/vt/cuda/cuda_gdn.cu, src/vt/cuda/gdn_prefill_conv.h, tests/vt/test_gdn_prefill_conv.cpp, tests/vt/test_ops_gdn.cpp, and pinned vLLM/FLA causal-conv plus fused post-conv kernels recorded with file:line anchors in the two spikes.

What changed

Add two independently selectable, default-off CUDA specializations. VT_GDN_POSTCONV_TOKEN_TILE=1 ports the upstream-shaped 16-token/four-warp post-conv schedule while preserving the existing 128-lane reduction tree byte-for-byte. VT_CONV_CHANNEL_TILE=1 specializes the causal-conv path for width four at the existing one-channel-per-thread grid; arm =2 is retained as the measured 256-channel/two-channels-per-thread falsification arm. Production dispatch and the portable mutation gate share one callback dispatcher, so the tests prove that each selected arm is actually invoked.

Evidence

scripts/agent-preflight.sh passes after rebasing onto canonical upstream/main c05cee1d2; all record gates, mutation suites, live-row audit, and committed-range document checks are green.
Tests covering this change: contained CUDA rebuild including an explicit vllm-bench relink; portable dispatch 9/9 cases, 88/88 assertions; CUDA GDN 67/67 cases, 4631/4631 assertions; Qwen3.5 paged-forward 4/4 cases, 8/8 assertions. The accepted production arms produce identical token hashes.
Same-change doc obligations: docs/STATUS.md and docs/BENCHMARKS.md updated; the model/feature surface did not expand. Full profiler evidence and the void stale-link series are retained in the two specs and .agents/benchmark-record.md.

Speed claims

The operator ran the numbers under ${GPU_LOCK} and they are recorded in docs/BENCHMARKS.md with the repro recipe.

On the local RTX 5070 Ti / Qwen3.5-4B c32 workload, the post-conv tile reduces graph-node time 227.887 to 122.587 ms (1.859x) and improves every observed enclosing axis, including total/output throughput by 0.532%. The causal-conv K4 arm reduces graph-node time 234.605 to 219.506 ms (6.44%) and improves whole-run total/output throughput by about 0.12%. The 256-channel arm is 4.05% slower than the K4 arm and is recorded as falsified.

Honest gaps

This PR is stacked on perf(gdn): dispatch exact causal-conv chunks #127. Until perf(gdn): dispatch exact causal-conv chunks #127 merges, GitHub shows its exact-chunks commits in this PR too; after that merge, this PR reduces to the opt-in follow-on commits.
Both specializations remain opt-in pending repeated A/B and hardware-unavailable 27B/35B gates. The 4B result is not extrapolated to the release models.
Same-tool residuals remain: post-conv is 1.135x and causal conv K4 is 1.509x slower than pinned vLLM.
The first production causal-conv profile series was void because vllm-bench had not relinked the rebuilt CUDA library. The accepted series explicitly relinked the benchmark and trace-proved the specialized kernels.
Current main c05cee1d2 independently fails check-device-leakage.py: src/vllm/v1/worker/gpu/runner.cpp contains one new hard-coded DeviceType::kCUDA (DSR kcuda 1 > baseline 0). perf(gdn): dispatch exact causal-conv chunks #127 and this PR inherit that merged-tree failure; neither kernel diff introduced it.

…re-push sandbox (#159) Two guards on main were RED and between them blocked every open external contributor PR (#127, #154, #155) and every push. Both premises were verified in the tree before changing anything. 1) check-device-leakage: src/vllm/v1/worker/gpu/runner.cpp named vt::DeviceType::kCUDA in the device-agnostic shared layer (DSR bucket 'kcuda' 1 > baseline 0). It came in with the QueueSupportsAsyncInputCombine rescope during the PR #140 fix round - ours, not the contributors'; richiejp reported it in #127's honest gaps. Fixed the way the guard's own message prescribes, mirroring the SupportsAuxStream precedent: ask the backend, not the device. New vt::Backend::SupportsAsyncSampledTokenReadback() (base false) answers whether the host may validly read the sampled token id back between steps; CPU overrides true (host and device memory are one allocation) and CUDA overrides true (the id is device-mirrored). The runner asks vt::TryGetBackend(queue.device.type), whose nullptr for a device absent from the build also subsumes the old #ifdef VLLM_CPP_CUDA guard. SEMANTICS UNCHANGED: CPU async-ON, CUDA async-ON, discrete non-CUDA (ROCm gfx1201) async-OFF - the "!"-token hazard stays closed. 2) .githooks/pre-push ran check-policy.py inside a PARTIAL export (README.md docs scripts .agents), but policy_contract.py:428 asserts AGENTS.md is a non-symlink regular file and resolves its Markdown links against that sandbox. AGENTS.md and its .env.example link were both missing, so the hook failed closed on content that is fine in the real tree - every push refused. EXPORT_PATHS is now a superset of what the CHECKERS read. Gates: check-device-leakage RED->GREEN (kcuda=0, DSR 32 == baseline 32); all four hook checkers OK in the reproduced sandbox; test_async_llm 8/8-347, test_engine_core 6/6-44, test_llm_engine 11/11-204 (CPU still resolves async-ON); clean -Werror CPU build; full 11-gate record battery green. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude (Opus 5) via Claude Code

Select the measured sm_120 post-conv residual, reject the existing split grid, and bind the upstream 16-token CUDA experiment and gates. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

Port the upstream 16-token per-head schedule behind an opt-in flag, preserve the existing reduction order byte-exactly, and record the sm_120 A/B evidence. Row: row/KERNEL-SSM-MAMBA-POSTCONV-TILE FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

Route runtime-width and both K=4 CUDA launch arms through the same portable, mutation-tested callback dispatcher. Pin strict arm-2 parsing and resolved non-K4 fallback semantics. row/KERNEL-SSM-MAMBA-POSTCONV-TILE FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

… op families + ForwardDevice (DGX GB10 gate 11/11·153) The last engineering brick before an actual DeepSeek-V4 run: the CUDA kernels for the four genuinely-new V4 op families, each a 1:1 device port of the landed portable HOST reference and unit-gated on the DGX GB10 against it at small shape. - MHC: Sinkhorn + mHC pre/post + hc_head collapse - DSA: indexer weight-fold + weighted-MQA ReLU logits + causal top-k + per-head attention-sink softmax + grouped output-LoRA - Compressor: softmax-window pool + RMSNorm + save-time APE + fp8_ds_mla KV encode (UE8M0 block scale + e4m3, bf16 rope) / decode - MoE: sqrtsoftplus/hash router + clamped SwiGLU New TU src/vt/cuda/cuda_deepseek_v4.cu (kernels + host-vector launchers + the OpProvider registration), registered through the vt seam under new OpIds kDeepseekV4{Mhc,Dsa,Compressor,Moe} and resolved by deepseek_v4_device.{h,cpp}. DeepseekV4Model::ForwardDevice runs the SAME W7 composition on the CUDA kernels via a V4Backend policy (DeepseekV4ForwardHost stays the host-ref oracle, unchanged). The 512-wide MLA attention + expert grouped-GEMM REUSE the existing NVFP4/FP8 kernels (cuda_mla_attn.cu, cuda_moe*.cu) and are NOT re-ported. Gate (DGX GB10, sm_121a): tests/vllm/models/test_cuda_deepseek_v4.cpp 11/11 cases · 153 assertions GREEN vs the host-ref oracle — BIT-EXACT ids (DSA causal top-k, sqrtsoftplus/hash router selection), -inf mask exact, near-tie rel-L2 < 1e-4 for the fp reductions, fp8_ds_mla round-trip within e4m3 granularity + bf16 rope bit-exact; PLUS the ForwardDevice composition gate (device == host, rel-L2 < 2e-3). compute-sanitizer memcheck 0 errors. RED-first proven (dropping the device sqrtsoftplus sqrt fails 3 cases / 6 assertions; revert restores). Also neutralized the pre-existing GCC-13 -O2 -Werror=array-bounds/-Wstringop- overflow FALSE POSITIVE in voxtral.cpp (project mudler#155) with a minimal local scoped #pragma GCC diagnostic around the in-bounds BuildPaddedDecodeAttn copies so the -Werror build links (advances mudler#155). SACRED-inert: shared MLA/MoE CUDA + the W3-W6 host-ref TUs are empty-diff; the host oracle test_deepseek_v4_forward stays 6/6·26. Honest 3-state: the kernels are RUNTIME-VERIFIED at small shape on real GB10; the real-checkpoint paged-engine e2e stays W8 (156.7 GiB NVFP4 does not fit ONE GB10), plus W2b tower materialization and the single-Spark IQ2_XXS-GGUF blk.N.* name-map. Records: kernel-matrix (+KERNEL-DSV4-W7-DEVICE, check-agent-record KERNEL 42->43), model-matrix (V4 row + checklist), coordination (CLAIM-DEEPSEEK-V4-W7-DEVICE), parity-ledger, state, docs/STATUS, docs/BENCHMARKS, roadmap_v1, spec §W7-device. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…re-push sandbox (mudler#159) Two guards on main were RED and between them blocked every open external contributor PR (mudler#127, mudler#154, mudler#155) and every push. Both premises were verified in the tree before changing anything. 1) check-device-leakage: src/vllm/v1/worker/gpu/runner.cpp named vt::DeviceType::kCUDA in the device-agnostic shared layer (DSR bucket 'kcuda' 1 > baseline 0). It came in with the QueueSupportsAsyncInputCombine rescope during the PR mudler#140 fix round - ours, not the contributors'; richiejp reported it in mudler#127's honest gaps. Fixed the way the guard's own message prescribes, mirroring the SupportsAuxStream precedent: ask the backend, not the device. New vt::Backend::SupportsAsyncSampledTokenReadback() (base false) answers whether the host may validly read the sampled token id back between steps; CPU overrides true (host and device memory are one allocation) and CUDA overrides true (the id is device-mirrored). The runner asks vt::TryGetBackend(queue.device.type), whose nullptr for a device absent from the build also subsumes the old #ifdef VLLM_CPP_CUDA guard. SEMANTICS UNCHANGED: CPU async-ON, CUDA async-ON, discrete non-CUDA (ROCm gfx1201) async-OFF - the "!"-token hazard stays closed. 2) .githooks/pre-push ran check-policy.py inside a PARTIAL export (README.md docs scripts .agents), but policy_contract.py:428 asserts AGENTS.md is a non-symlink regular file and resolves its Markdown links against that sandbox. AGENTS.md and its .env.example link were both missing, so the hook failed closed on content that is fine in the real tree - every push refused. EXPORT_PATHS is now a superset of what the CHECKERS read. Gates: check-device-leakage RED->GREEN (kcuda=0, DSR 32 == baseline 32); all four hook checkers OK in the reproduced sandbox; test_async_llm 8/8-347, test_engine_core 6/6-44, test_llm_engine 11/11-204 (CPU still resolves async-ON); clean -Werror CPU build; full 11-gate record battery green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

@richiejp

…ile + K=4 causal-conv arm (mudler#155) External contribution from @richiejp. Tracked by mudler#206, now linked in the roadmap issue table and in both specs (it was in neither). The kernels reviewed clean: no OOB index, no shared-memory overrun, no divergent barrier, no missing tail guard, and no launch-config assumption the launcher does not guarantee. The token-tile reduction-identity claim is real and was verified bitwise -- the qsum = (q0^2+q2^2)+(q1^2+q3^2) fold is exactly the shipped 128-lane tree's first two levels for the lane + item*32 mapping, 0 of 20,000 random f32 rows differ. The portable-header suite is genuinely strong: 9 cases / 88 assertions green with 8 of 8 mutations caught. FIXED WHILE LANDING -- the added token-tile test could not catch the defect the author had to fix. RunGdnPostConvTokenTileCase hardcoded bf16 for q/k/v_out and asserted CHECK(q1 == q0) byte-exact. The spec records that the FIRST warp-reduction ordering tried (summing the four lane-owned squares sequentially) CHANGED production tokens and was rejected for it. Re-creating that rejected arithmetic and pushing it through the bf16 store differs on 0 of 11,160 rows and 0 of ~1.4M elements: the f32 sums do differ (3,580 of 20,000 rows), but 1/sqrtf moves ~1 ulp f32 and __float2bfloat16 absorbs it entirely. bf16 keeps 8 mantissa bits, so a bf16 byte-exact assertion is a tolerance gate wearing an exactness costume, and it was satisfied by an implementation already known to be wrong. The case now sweeps qkv over {f32, bf16}, f32 first; the f32 arm fails the mutant on 3,580/20,000 rows. Only a manual e2e token-SHA on sm_120 caught this before, which is neither in CI nor reachable by a reviewer. Conflict resolution, per the keyed-record rule: - .agents/benchmark-record.md is the one genuinely append-only log here, so both sides were UNION-APPENDED in chronological order -- main's 35B mid-band section, then this row's two 2026-08-08 sm_120 sections. No content from either side dropped. - .agents/NOW.md is a keyed record and was NOT auto-merged. Taken from origin/main wholesale, then exactly three scoped edits reapplied (the header work line, the Qwen3.5-4B sm_120 row, numbered item 3), each asserted to match a unique anchor. Main's newer Kimi-Linear-48B, 35B mid-band and 27B NVFP4 rows were verified byte-present afterwards. The branch's stale GPU-lock bullet (flock /tmp/gpu) and operator-protocol rewording were deliberately NOT carried: main is newer on both, and the former contradicts the local-ai-worker parking convention. - docs/STATUS.md, docs/BENCHMARKS.md, .agents/roadmap_v1.md auto-merged on disjoint keys; every main-side key (27B, NVFP4, 35B, Qwen3.6, issue rows mudler#287/mudler#285/mudler#213) was then verified present by count rather than trusted. Also fixed: - The red agent-record check was a stale docs/STATUS.md character ratchet: the branch shrank the page 137 chars without lowering STATUS_RATCHET["chars"]. Set to 243405, MEASURED after resolution (len of the resolved file), not predicted. Strictly down, the only direction it may move. check-public-doc-tables and its 53 unit tests are green. - docs/BENCHMARKS.md quoted a 233.955 ms baseline against the 219.506 ms arm-1 figure, which yields 6.18% and not the 6.44% stated everywhere else; 219.506 was measured against 234.605 in the SAME run. Now quotes the same-run arm-0 basis and says so. NOT fixed here, filed instead as mudler#305: the exact-chunk mapping has only chunk 0 read the initial conv_state while only the last chunk writes the final one -- different, unordered blocks, so they race whenever has_initial_state is set and t_len > kConvExactM. I confirmed both sites in cuda_gdn.cu on origin/main independently of the review. It is default ON (VT_CONV_REG + exact chunks) and predates this row, so it is a bug found while doing something else and gets its own issue, not a silent fix. Upstream causal_conv1d.py:147-150 keeps the read and the tl.store in the SAME program with a debug_barrier between. The channel-tile spec claimed preloading both stripes was the fix; corrected to say it narrows the window without closing it. Evidence boundary: this box has no GPU and no nvcc, so no CUDA kernel or test in this row was executed by me. The portable-header results above were run; the CUDA-side reduction claims were verified by faithful host emulation over the same index space and by bitwise arithmetic replay, which is transcription rather than compiled-kernel evidence. The modified test compiles clean and skips without CUDA; it still owes a real GPU run. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

@richiejp

…ler#266) External contribution from @richiejp, tracked by mudler#206. Sibling of mudler#155 by the same author; the two are INDEPENDENT branches over the same kernels, not a chain, so this merge reconciles them rather than stacking them. Every check on the PR was green except the pre-existing main baseline sanitize-cpu (address,undefined). Despite the title there is no sm_120-specific code in the diff -- no __CUDA_ARCH__, no capability query, no cmake change; "sm_120" names the box it was measured on. cmake/CudaArchFeatures.cmake is untouched, so the fp4-mma / cutlass-nvfp4 / cutlass-fp8 / marlin-nvfp4 / fa2 routing cannot have silently dropped a capability -- that historical failure mode is structurally out of scope here. What lands: three default-OFF env-gated CUDA experiment arms (a compile-time K=4 causal conv, a 16-token per-head post-conv schedule, and BV16 + shared-bank swizzle + register-resident-state arms of the fused GDN decode), a CUDA-free launch-geometry header so selection logic is unit-testable off-GPU, and an atomic multi-request admission path (add_request_wave -> add_requests_async -> put_many_nowait) that removes the frontend/batching confound from the bench. Default-path equivalence was traced rather than assumed: unset env resolves decode to <kBv32, swizzled=false, regstate=false> with sdk = dk+1 and the same else-branch as before; conv to kRuntimeWidth with chan_tiles = (c+127)/128, identical to the deleted arithmetic; post-conv to the unchanged fast predicate and grid. BlockingQueue gained a DEFAULTED template parameter, which cannot change behavior and breaks no forward declaration. FIXED WHILE LANDING: - The token-tile post-conv test recurs mudler#155's weakness, so the same repair is reapplied on this branch's copy: RunGdnPostConvTokenTileCase now takes a DType and sweeps {f32, bf16}, f32 first. bf16 keeps 8 mantissa bits, so __float2bfloat16 absorbs the ~1-ulp f32 movement a reassociated q/k norm produces -- a bf16-only byte-exact assertion is a tolerance gate wearing an exactness costume, and an independent CPU re-creation of the kernel's arithmetic put the catch probability for a subtle reassociation at ~11%. - The page published TWO cross-engine results for ONE key: 1.0283x in the STATUS row and 1.021246x twice in the prose below it, with BENCHMARKS carrying a third pairing. Collapsed to a single standing result -- the corrected three-repetition run -- naming 1.021246x explicitly SUPERSEDED (it predates the atomic wave admission that removed the timing confound) rather than leaving two live numbers to choose between. - The campaign spec still records the run as an implementation checkpoint whose mutation re-review, operator gate and real-GPU token identity are owed, while four public surfaces stated the result flatly. The spec is the authority, so STATUS and BENCHMARKS now carry an explicit PENDING disposition. Exactly one result per rule: this one is pending, not passing. Conflict resolution (13 files), per the keyed-record rule: - src/vt/cuda/*, tests/vt/test_gdn_prefill_conv.cpp, tests/vt/test_ops_gdn.cpp taken from this branch after confirming it is a strict SUPERSET of what mudler#155 landed: both trees define the same 43 kernels, and this one adds VT_GDN_DECODE_BV / _SWIZZLE / _REGSTATE. The F32 fix was then reapplied. - .agents/benchmark-record.md union-appended (genuinely append-only). - .agents/NOW.md, roadmap_v1.md: main taken wholesale, only the scoped Qwen3.5-4B edits reapplied; main's newer Kimi-Linear-48B and 35B mid-band rows verified byte-present afterwards, and this branch's stale copies of them deliberately not carried. - .agents/kernel-matrix.md kept BOTH dated checkpoints (2026-08-08 from mudler#155, 2026-08-09 from this row) in chronological order. - docs/ENVIRONMENT.md took this branch's block (the only side documenting VT_GDN_POSTCONV_TOKEN_TILE) with mudler#155's richer VT_CONV_CHANNEL_TILE text restored. - The two ratchets (mudler#155's 243405, this branch's 243519) reconciled into one MEASURED value, 243455, at both the checker and the test ceiling. Strictly down, the only direction it may move. The prose collapse pays for its own explanation out of the page rather than adding to it. Known debt recorded, not hidden: add_request_wave is a shipped engine capability reachable only from examples/bench and absent from include/vllm.h; VT_BENCH_PRETOKENIZE defaults ON, so historical BENCHMARKS rows for 27B / 35B / DeepSeek / MXFP4 were produced on the old frontend and are not directly comparable (VT_BENCH_PRETOKENIZE=0 is the same-binary rollback). The new K4 kernel inherits the conv_state cross-block race filed as mudler#305; it is opt-in and the structure is byte-identical to the shipped kernel, so this is inherited exposure, not new. Evidence boundary: no GPU and no nvcc on this box, so no CUDA kernel or test here was executed. Default-path equivalence, launch geometry and the swizzle bank mapping are static reading of the diff plus hand-evaluated index arithmetic. The tree builds clean on CPU (422/422) and the doc-table checker plus its 53 unit tests are green. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

main advanced by ten commits while the six external PRs were being reviewed and merged (SILU-VECTORIZE, SERVE-METRICS-ASYNC mudler#277, SAMPLE-LOGPROBS-RECORD-REPAIR mudler#259, PERF-27B-LMHEAD-DSR mudler#298, PERF-35B-SHARED-DOWN-BF16, ENG-OPERATOR-RECORD- COVERAGE-FIX mudler#304, release mudler#303). Resolved rather than re-based so nothing of either side is lost. - .agents/roadmap_v1.md: issue rows are DISTINCT keys, so the two sides union -- main's mudler#298 alongside this landing's mudler#269/mudler#305/mudler#206. - .agents/NOW.md: main's 35B mid-band row is NEWER (two levers landed, not one) and was taken wholesale; only the Qwen3.5-4B row was reapplied on top, with main's row verified byte-present afterwards. The "next actions" hunk kept this side's item 3, which post-dates main's copy. - .agents/benchmark-record.md union-appended, main's entries first. - tests/vllm/v1/test_async_llm.cpp: both sides add a distinct `using` declaration (PublishAsyncRequestWaveIfAlive from mudler#266, PrometheusStatLogger from mudler#277) -- union, not a choice. - scripts/check-public-doc-tables.py: main had re-pinned the STATUS ratchet to 243512 for mudler#277 while this landing had reconciled mudler#155 and mudler#266 to 243455. Both rationale histories are kept and the value re-MEASURED on the merged page: 243368. Strictly down, the only direction it may move; the byte-tight test asserts equality, so this is measured, never predicted. Doc-table checker and its 54 unit tests green; clean CPU build. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…nel-tile spike .agents/specs/sm120-qwen35-conv-channel-tile-2026-08-08.md carries four raw conflict markers on main (lines 4/7 and 144/160, tagged `pr-266`), left behind when mudler#266 and mudler#155 landed. Both hunks have an EMPTY incoming side, so the resolution is to keep the HEAD text and drop the markers: the issue/PR attribution line, and the paragraph recording that the arm-2 preload NARROWS the stripe-overwrite window rather than closing it, with the residual hazard tracked as mudler#305. That second paragraph is why this is a commit and not a silent tidy: it is the spike's honest-limits statement, and a reader who stops at the `<<<<<<<` line never reaches it. No content is added or removed beyond the four marker lines. Refs mudler#266, mudler#155, mudler#305. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…ng (mudler#267) Merges `row/SAMPLE-LOGPROB-TOKEN-IDS` `d6b482b6`, closing mudler#264. Spec `.agents/specs/logprob-token-ids.md`, committed in its own commit before any implementation code. Row `SAMPLE-LOGPROB-TOKEN-IDS` stays `PARTIAL`. WHAT IT DOES. A request may now name an EXPLICIT set of vocab ids and get back exactly those plus the sampled token -- vLLM's generative-scoring path (`generative_scoring/serving.py:247-255` sends `max_tokens=1` + `logprob_token_ids=label_token_ids`), and the efficient answer to "score these five labels" without `logprobs=-1` and a full-vocab sort. Ported 1:1 at pin `555967922`: the field and `MAX_LOGPROB_TOKEN_IDS` (`sampling_params.py:31, 278-283`), the `num_logprobs` PROPERTY (`:724-729`), the two config-free validations (`:773-782,795-801`), req-id-keyed `InputBatch` tracking re-keyed to req-index over the live batch (`gpu_input_batch.py:273,443-444,574,934-951`), `gather_specific_token_logprobs` (`sampler.py:151-225`) with the padded `[n, max+1]` row, sampled token in column 0, `-inf` padding and the sampled token's rank over the FULL vocab, the snapshot condition (`:86`) and the precedence rule that explicit ids WIN over a count (`:133-136`). THE HIDDEN HALF, which is what the review loop surfaced. Three consumers spelled upstream's `num_logprobs` PROPERTY as the raw `logprobs` field -- the scheduler's slice gate (`scheduler.py:1818`), `LogprobsProcessor::FromNewRequest` (whose comment already CLAIMED it was the property) and `RequestState::FromNewRequest`. Identical for every request that exists today, so nothing was visibly broken; without fixing them a scoring request produced sampler output that nothing downstream ever read and the feature was unreachable through the engine. A SECOND RED landed mid-implementation for exactly that reason: with the sampler and `InputBatch` fully wired, `test_llm_engine` was still red because `RequestState::FromNewRequest` gated the whole `LogprobsProcessor` on `sp.logprobs.has_value()`, where upstream constructs it unconditionally (`output_processor.py:223-229`). Our device-resident greedy fast path is OURS, not upstream's, and skips the gather entirely; it is now gated on the same combined predicate, and missing that second edit would have made the feature vanish silently on the async greedy path. Issue mudler#249's defect class is kept out (every requested id bounds-checked into `[0, vocab)` and every map key into `[0, n)`, mirroring what `torch.gather` raises on) while its instance, `GatherLogprobs`' unbounded `k`, is deliberately untouched. The branch's own main-merge found three PRODUCT conflicts with mudler#238 (`logprobs_mode`) and mudler#223 (`prompt_logprobs`), which landed after it was written, and resolved all three by keeping BOTH features -- the mode selects WHICH tensor the snapshot holds, the ids select WHICH entries are read out of it. It added ONE test neither PR's base could have carried, "logprob_token_ids reads the PROCESSED snapshot under a processed mode", RED-first proven by mutation (weakening the request to `num_logprobs.has_value() && processed_mode` leaves 20 of 21 cases green and fails only this one), tree restored byte-for-byte. KEYED RECORDS -- main's version taken WHOLESALE, the branch's scoped edit reapplied by hand, every deleted anchor asserted to occur exactly once, and every non-reconciled path proven byte-identical to the branch's own edit set. This is the THIRD of three merges in one landing, so several records had already moved under PR mudler#324 and PR mudler#282 and none of the branch's own measurements survived. docs/STATUS.md auto-merge accepted only after proving the edit set matches the branch's exactly; the Sampling row and mudler#282's LoRA row are different rows. Merged page RE-MEASURED: 243,119 = 243,128 after mudler#282, less the Sampling row's own net -9. STATUS ratchet `scripts/check-public-doc-tables.py` CONFLICT for the second time in this landing. ALL rationale histories kept, append-only: main's, mudler#282's, and mudler#267's. Pinned to the RE-MEASURED 243119. The branch's 243278 was DISCARDED -- it was measured against `5812b8b6`, before main's own re-pin AND before mudler#282's collapse. Byte-tight (`test_the_rebased_character_ratchet_is_byte_tight` requires cap == len(page) exactly), strictly DOWN 243188 -> 243128 -> 243119 across the landing. ratchet CEILING `tests/scripts/test_check_public_doc_tables.py` CONFLICT. Lowered to 243119 in the SAME change as the ratchet; both branches' stale ceilings (243482, 243479 -- both computed against `5812b8b6`) discarded, and both collapses' rationale carried in the comment. EVERY mutation guard from every side is kept: main's `test_the_ratchet_is_exactly_one_byte_wide` and `test_a_repin_can_only_tighten_the_char_ratchet`, this branch's NEW `test_one_char_of_growth_on_the_LIVE_page_is_rejected`, the byte-tight guard, the no-hidden-headroom guard, the only-ever-moves-down guard, and BOTH paid-for-by-a-real-collapse guards (mudler#223's beam-search collapse and mudler#238's best_of rationale). The branch could not re-spend mudler#223's collapse, so it collapsed FOUR different restatements in the same Sampling cell instead -- each a definition of what an OpenAI field DOES rather than a statement of what we support -- and all three substrings those two guards pin are intact. 59 tests, all pass. docs/BENCHMARKS.md the branch's terse 145-byte `logprob_token_ids` row landed inside the headroom PR mudler#282's merge bought by moving the superseded 2026-08-08 `BENCH-VK-LLAMA` row into `.agents/benchmark-record.md`. No cap was raised and nothing else moved for this merge: 44,709 -> 44,854 of the hard 45,000. .agents/NOW.md CONFLICT. Main + mudler#282 carried the `logprobs_mode` (mudler#238) row and mudler#282's compacted `Surface coverage` cell and its new `LORA-RUNTIME` row; this branch UPDATES the mudler#238 row in place into `SAMPLE-LOGPROB-TOKEN-IDS` rather than adding a second one. All three kept. The net +19 took the page to 6,005 over the hard 6,000 budget, so it was paid for inside the page: the `Work:` line still announced the PREVIOUS landing and now names this one, and the TP spike row's "(unblocks mudler#127/mudler#154/mudler#155)" clause is stale -- mudler#154 and mudler#155 are MERGED and mudler#127 is CLOSED, verified with `gh pr view`. 5,978 chars / 94 lines. .agents/engine-matrix.md the `SAMPLE-LOGPROB-TOKEN-IDS` row auto-merged into main's file and was verified line-for-line against the branch's scoped edit. The row's State does NOT move (`PARTIAL` before and after -- mudler#238 already moved it off `INVENTORIED`), only its Owner gains `CLAIM-SAMPLE-LOGPROB-TOKEN-IDS`, so the lifecycle rollup is deliberately untouched here and stays at the totals PR mudler#282's merge RECOMPUTED. `check-agent-record` confirms ENGINE=147. .agents/coordination.md the branch's prose claim only; the claims TABLE that `check-agent-record` cross-references is byte-for-byte main's plus mudler#282's `CLAIM-LORA-RUNTIME-W2` row. .agents/roadmap_v1.md CONFLICT. mudler#282's `mudler#278` issue row and this branch's `mudler#264` row are distinct keys, unioned. The C7 portfolio row's stale gap list is corrected by the branch's own scoped edit. .agents/porting-inventory.md `logprob_token_ids` leaves the deferred-stub list; the same sentence's stale `logprobs_mode` entry is corrected in passing because this edit rewrites it. docs/USAGE.md the new field with an example; mudler#282's LoRA paragraph sits above it and both survive. .agents/benchmark-record.md untouched by this branch beyond the row PR mudler#282's merge moved in; it is the one genuinely append-only log in this landing. RESIDUALS, which is why the row is `PARTIAL`: the `logprobs_mode` variants (open PR mudler#258 owns them), the OpenAI request field on `/v1/completions`, `/v1/chat/completions` and `/v1/generative_scoring`, and vocab-range validation in `Verify()` -- which has no model config, exactly as for `allowed_token_ids`, and the sampler bounds the ids anyway, so that one is message quality, not safety. GATE, re-run by the operator on the merged tree, CPU Release, foreground, unbounded: cmake --build build-cpu -j 18 834 targets, 0 errors ./build-cpu/tests/test_sampler 21/21 cases, 114 assertions, 0 skipped ./build-cpu/tests/test_input_batch 29/29 cases, 205 assertions, 0 skipped ./build-cpu/tests/test_llm_engine 24/24 cases, 492 assertions, 0 skipped scripts/check-agent-record.py OK, ENGINE=147 MODEL=362 QUANT=82 KERNEL=51 BACKEND=80 scripts/check-public-doc-tables.py OK scripts/check-now-current.py OK scripts/check-fusion-consistency.py OK, 0 drift tests/scripts/test_check_public_doc_tables.py 59/59 `scripts/__pycache__` was cleared before every checker run: the ratchet values in this landing are the same byte length as the ones they replace, so a stale `.pyc` survives mtime/size invalidation and a checker will silently keep reading the old number. `test_llm_engine` reads 492 assertions here against the 493 the branch measured on its own base. Both are 24/24 cases, 0 failed, 0 skipped -- no case is unreached, so this is not a killed run. The delta is main's, not this merge's: `origin/main` changed sixteen files under `src/vllm/v1/engine` and `include/vllm/v1` between `5812b8b6` (the branch's base) and `91763643`, and its only edit to the test file itself was to ADD five lines. The FULL `ctest -j 6` for this landing is reported on the reconciliation merge that follows this one, because `origin/main` advanced to `75a29016` while these three were being gated and the binding gate belongs on the tree that is actually pushed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…retired under us `origin/main` advanced a SECOND time during this landing, 31 commits, while the reconciliation merge below was being gated. A plain `git push` was refused as non-fast-forward, which is git protecting those merges, so this fetches, re-merges, reconciles every record against the shape main now has, and re-gates. No force, no rebase of a published branch. THE TWO CONSTRAINTS THIS LANDING WAS BUILT AROUND NO LONGER EXIST. mudler#364/mudler#368 (`ENG-RECORD-CONFLICT-SURFACES`) retired them as a defect, and AGENTS.md gained the rule behind it: *no surface that every PR must write*. Concretely: - `STATUS_RATCHET["chars"]` is DELETED from `scripts/check-public-doc-tables.py`. It was a byte count of one file stored in another and allowed only to fall, so every PR owing STATUS.md a line had to evict unrelated prose AND edit the checker. The three QUALITY keys (`h2_sections`, `long_paragraphs`, `oversized_cells`) are kept. - `MAX_CHARS` is DELETED from `scripts/check-now-current.py`. NOW.md was a fixed-size shared buffer at exactly 6000/6000, so adding a row meant evicting someone else's. `MAX_LINES` and `MAX_ENTRY_CHARS` are kept, and they cap the ENTRY rather than the file. So the ratchet arithmetic this landing carried out three times -- 243188 -> 243128 -> 243119 -> 243117, each byte-tight and re-measured -- is now moot, and the honest resolution is to DROP it rather than defend it. Main's version of both checker files and of `tests/scripts/test_check_public_doc_tables.py` is taken BYTE-FOR-BYTE, which also drops PR mudler#267's `test_one_char_of_growth_on_the_LIVE_page_is_rejected`: that guard asserts on the `chars` key, and the key is gone. Its subject was retired, not its argument. `grep 'STATUS_RATCHET\["chars"\]'` over `scripts/` and `tests/scripts/` returns nothing. Equally, the NOW.md compactions this landing made to buy room -- the `SERVE-METRICS` restatement, the MiniMax-H3 and Release cells, the TP row's `(unblocks mudler#127/mudler#154/mudler#155)` clause -- are reverted to main's fuller text: they were payments for a budget that no longer exists, and carrying them would be unrelated churn in a keyed record. Also fixed on main and no longer owed by anyone: the four conflict markers committed to `.agents/specs/sm120-qwen35-conv-channel-tile-2026-08-08.md`, which this landing found on main and did not touch (`76dfe8dc`). KEYED RECORDS, reconciled against the shape main now has. scripts/check-public-doc-tables.py, tests/scripts/test_check_public_doc_tables.py main's version taken BYTE-FOR-BYTE (`git diff` vs `f64f2b71` is empty for both). The landing's ratchet work is dropped entirely, per above. .agents/roadmap_v1.md CONFLICT. Main SORTED the whole issue table by Row then issue number, which moved every line. Main's sorted table taken wholesale and the landing's SIX rows re-inserted in sorted position -- mudler#299, mudler#314, mudler#337, mudler#338 under `ROAD-V1-C1`, mudler#278 under `LORA-RUNTIME`, mudler#264 beside mudler#238 under `SAMPLE-LOGPROB-TOKEN-IDS`. Verified: all 48 of main's rows present, exactly 6 added, and the only other line that differs from main is the `ROAD-V1-C7` portfolio row, which PR mudler#267 deliberately updates. .agents/engine-matrix.md CONFLICT. Main's `Serving, API, CLI, library` row gained a row and a `READY` (27 -> 28). Main's row kept wholesale, the landing's `LoRA and adapters` row (`ANCHOR-BACKFILL` -> `ACTIVE`) reapplied, and the **Total** RECOMPUTED from the ten actual area rows rather than carried from either side: 148|35|16|4|9|27|8|9|39. `check-agent-record.py` confirms ENGINE=148. .agents/coordination.md auto-merged and verified byte-identical to main's edit set. mudler#368 introduced `.agents/claims/` (one file per claim) but deliberately does NOT migrate existing rows -- the checker still reads the legacy table -- so PR mudler#282's `CLAIM-LORA-RUNTIME-W2` row stays where it is and is removed when the claim closes. scripts/check-gate-commands.py CONFLICT: both sides added a comment block at the same anchor. Unioned; `RUNNABLE_BASELINE` now carries `ENG-RECORD-CONFLICT-SURFACES` (main's), `ENG-RELEASE-CONTAINERS` (main's) and `LORA-RUNTIME` (mudler#282's). 117 gated rows, 32 runnable. docs/FEATURES.md CONFLICT. Main's newer prose wins (37 registered architectures, up from 35) and PR mudler#324's merged-GEMM sentence is reapplied into it. First attempt as a separate paragraph tripped the 21-vs-20 prose-paragraph cap and appending it inline tripped the 700-char paragraph cap at 782, so it is paid for INSIDE the paragraph exactly as mudler#324 originally did -- the lead-in collapsed, total 697 of 700. No cap was raised. .agents/NOW.md CONFLICT. Main's three fuller rows taken wholesale (`BACKEND-ROCM` now records the gfx1100 GDN slice and mudler#269 M0-M4). 94 of 100 lines, every entry inside MAX_ENTRY_CHARS. docs/STATUS.md, docs/BENCHMARKS.md untouched by main this time; 243,117 and 44,839. BENCHMARKS is still 161 chars inside its hard 45,000 cap thanks to the row PR mudler#282's merge moved into `.agents/benchmark-record.md`. GATE for the WHOLE landing, re-run by the operator on THIS tree, CPU Release, foreground, no timeout on any test binary. cmake --build build-cpu -j 18 692 targets, 0 errors, 0 warnings Focused, all seven declared gates of the three PRs: test_dense_gate_up_seam_forward 4/4 cases | 1940 assertions | 0 skipped test_linear_method 6/6 | 76 | 0 skipped test_lora_layers 16/16 | 4498 | 0 skipped test_punica_cpu 8/8 | 149 | 0 skipped test_sampler 21/21 | 114 | 0 skipped test_input_batch 29/29 | 205 | 0 skipped test_llm_engine 24/24 | 494 | 0 skipped Checkers, `scripts/__pycache__` cleared before each: check-agent-record.py OK ENGINE=148 MODEL=362 QUANT=82 KERNEL=51 BACKEND=80 check-public-doc-tables.py OK BENCHMARKS 44,839 / 45,000 FEATURES 27,371, longest prose 697 / 700 check-now-current.py OK 94 / 100 lines check-fusion-consistency.py OK glue 14 / 12 routed / 2 allowlisted; merged-gemm 10 / 6 / 6, 0 drift check-gate-commands.py OK 117 gated rows, 32 runnable check-commit-trailers.py OK over the whole landing range Checker unit suites, all seven: test_check_public_doc_tables 52/52, test_check_gate_commands 28/28, test_check_fusion_consistency 20/20, test_agent_record 29/29, test_check_now_current 11/11, test_record_merge_shape 11/11, test_check_pr_size 38/38. Full `ctest --test-dir build-cpu -j 6 --output-on-failure`, 383 tests, 2771.26 s. BOTH numbers reported: 99% tests passed, 2 failed out of 383 -- `test_async_llm` (0.33 s) and `test_openai_conformance` (163.91 s). Both are on the known starvation-prone list, and both pass SERIALLY on an idle box (load 2.52): test_async_llm 15/15 cases | 443 assertions | wall 0.05 s test_openai_conformance 23/23 | 252 | wall 0.45 s `test_openai_api_server`, which needed a serial re-run last time, PASSED under -j 6 here in 128.64 s -- the same binary, the same tree, a quieter box, which is the clearest available evidence that these are scheduling artifacts rather than defects. Neither failing binary loads a file this landing touches. The mudler#274 ASan/UBSan five did not appear: this gate is Release with no sanitizer, and `test_llm_engine`, `test_capi` and `test_llama_embedding_fold` -- three of that five -- all PASS here (2006.17 s, 1618.64 s, 14.20 s). Note on `test_llm_engine`'s assertion count, corrected earlier in this landing: it is run-to-run NONDETERMINISTIC in this binary (493, 493, 492 measured across three consecutive runs of one unmodified build; 494 here). Every run is 24/24 cases, 0 failed, 0 skipped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

richiejp force-pushed the row/KERNEL-SSM-MAMBA-POSTCONV-TILE branch 3 times, most recently from b3f019a to a651838 Compare August 10, 2026 13:56

richiejp added 6 commits August 10, 2026 15:11

docs(gdn): spike causal-conv channel tiling

46a407c

FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

perf(gdn): add causal-conv channel tile arms

d6c3ec7

FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

record(gdn): select K4 causal-conv specialization

7c8ee10

FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Codex:gpt-5.6-sol [Codex]

richiejp force-pushed the row/KERNEL-SSM-MAMBA-POSTCONV-TILE branch from a651838 to 7c8ee10 Compare August 10, 2026 14:12

localai-bot merged commit 19e3554 into mudler:main Aug 11, 2026
11 of 13 checks passed

Navigation Menu

perf(gdn): optimize post-conv and causal-conv kernels - #155