feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA prefill (RDNA4) - #228

bakon11 · 2026-08-09T23:34:05Z

Summary

Gemma-4-26B MoE ROCm path for dual-GPU (gfx1201 lab):

Multi-EOS from config.json ∪ generation_config.json ([1,50,106])
Host expert LRU + dual-GPU native FP8 resident packs
Fused ExpertGeGLU FP8 decode (M=1)
Prefill: group-by-expert batch (T≥64), FP8 channel GEMM M>1 (weights stay FP8 on GPU), device gather + weighted scatter (no host weight dequant / no host hacc)
Portable vt::fused_ops seams — no vt::rocm from models/

Lab (2× R9700, ROCm 7.2.4)

Prompt	Prefill tok/s	Quality
~800	~167	READY
~6k	~123	READY
~10k	~192	READY
Paris warm	—	Paris

hipBLASLt W8A8 microbench still slower than BF16 GemmEx on gfx1201 for pure matmul — not default.

Size

Over the ~900 product-line ratchet (MoE + HIP kernels). Happy to split further if maintainers prefer (decode/resident vs prefill batch).

Supersedes

Supersedes the bulk of #205 content (rebased to current main as a single tip commit). Please close #205 in favor of this + the serve reliability PR.

Test plan

check-device-leakage
check-env-doc / FEATURES / USAGE / ENVIRONMENT
Live :8010 dual FP8 resident smoke (Paris/READY/prefill rates)
CI

bakon11 · 2026-08-10T00:45:26Z

Update after live Hermes proof (RDNA4 / dual R9700):

Full agent path works end-to-end (SOUL ~38k prefill → streamed decode → real reply). The blocking EngineCore fatals after prefill were missing ROCm V1 sampler ops (op 30 temperature, op 40 allowed_token_ids) + Hermes max_tokens=-1.

That fix is split out as a small tip-of-main PR for fast merge:
→ #234

This #228 remains the larger FP8 resident / fused decode / GPU prefill-MoE package. Prefer landing #234 first, then rebase this head.

bakon11 · 2026-08-10T00:50:49Z

TESTERS WANTED — clear split: FP8 (lab) vs BF16/FP16 MoE (community)

What this PR is

FP8-first Gemma-4 MoE on ROCm: native FP8 resident experts, fused FP8 decode helpers, GPU prefill-batch MoE, etc. Lab-proven on 2× R9700 gfx1201 with Hermes-scale prompts (slow decode at 38k, but correct).

What it is not

A claim that every kernel here is the BF16 path. FP8 weight GEMM / FP8 resident upload do not apply to BF16 weights.

What does still matter for BF16/FP16 MoE testers

If you run BF16 or FP16 MoE on ROCm, you still want:

Land first	Why
#234	Full V1 sampler on ROCm + Hermes `max_tokens=-1` (EngineCore `op 30`/`op 40` fatals are dtype-agnostic)
#227	KV admit fail-fast + clamps (large-prompt hangs are dtype-agnostic)

Optional on this PR after those: any shared MoE control flow you can exercise in BF16 (group-by-expert prefill batch, dual-GPU placement) — report whether BF16 hits the same batch path or stays on serial expert loops.

Please do not treat as MoE validation

Dense Gemma-4-12B BF16 — no experts

Ideal tester hardware

BF16/FP16 Gemma-4-26B MoE (or similar) checkpoint
Multi-GPU ROCm with headroom beyond ~52 GiB weights
Host RAM ≫ 30 GiB preferred for clean loads

Report template

GPU / ROCm / dtype / model
Which PR builds (feat(rocm): V1 sampling ops for Hermes EngineCore #234 / fix(server): fail-fast unschedulable KV waits #227 / this)
Short + long prompt results
Prefill batch on/off if visible in logs (prefill-batch / serial)
Crashes: no kernel for op, OOM, hang

Prefer merge order: #234 → #227 → rebase this FP8 MoE package.

Thanks — FP8 is the consumer-RDNA4 daily driver; BF16/FP16 MoE needs wider silicon than this lab can soak alone.

localai-org-maint-bot

The lab evidence is useful, but this introduces a large FP8 resident/prefill execution surface without automated kernel/model parity tests; the only added tests cover HF config parsing. Please add focused parity coverage for resident expert placement, fused decode, and grouped prefill (including multi-device behavior or a testable dispatch seam), then rebase after #234/#227 as proposed and clear the current repository gates. The current single-hardware smoke is not sufficient protection for this execution-path change.

…config.json `InputProcessor::generation_config_eos_ids_` is named for generation_config.json but was populated entirely from config.json's `eos_token_id`. Nothing in the LLM path read generation_config.json at all (only Parakeet's loader did), so upstream's "secondary eos ids" were, in our port, just a second copy of the primary list. Upstream loads both. ModelConfig.try_get_generation_config (vllm/config/ model.py) reads generation_config.json whenever --generation-config is "auto" (the default) or "vllm", and SamplingParams.update_from_generation_config (vllm/sampling_params.py:645-655) merges its ids into stop_token_ids. The two files routinely disagree: Gemma-4-26B ships config.json eos: [1, 106] generation_config.json eos: [1, 106, 50] so id 50 — a turn-level stop — never terminated a generation for us. The symptom is a chat model that runs past its stop token to the length cap. Found via mudler#228, which hit this on Gemma-4 and fixed it by merging the ids into `cfg.raw["eos_token_id"]`. That lands them at the wrong layer: raw's `eos_token_id` supplies the PRIMARY eos id, which our existing 1:1 port of update_from_generation_config already gates on `ignore_eos`, while these are SECONDARY stop ids. Rewriting raw would make them unconditional, and — since the primary is `generation_config_eos_ids_.front()` on the list path — a re-sorted union could silently displace which id is primary. So this carries the field separately (`HfConfig::generation_config_eos_ids`), leaves `raw` untouched, and unions by append so the primary keeps its position. The existing ignore_eos gates then apply unchanged: the ids reach stop_token_ids only when ignore_eos is false, and all_stop_token_ids (min_tokens masking) regardless — which is exactly sampling_params.py:653. Red first: both new suites fail 2 assertions each against the current implementation and pass with it. A malformed or absent generation_config.json is a silent no-op, mirroring try_get_generation_config returning {} rather than raising. test_hf_config 194/194 (+5 subcases) test_input_processor 64/64 (+2 subcases) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]

bakon11 · 2026-08-10T13:08:54Z

Status (lab + PR hygiene)

Land order: #234 (ROCm V1 sampler, rebuilt tip-of-main, clamp dropped) → #227 (KV fail-fast, rebuilt tip-of-main, prefix probe + clamp dropped) → then this #228 rebase.

Maintainer notes absorbed for the next #228 push:

Drop hf_config / FEATURES eos hunks (landed as fb2dfac7 / fix(parity): read generation_config.json eos ids — we only ever read config.json #262)
Drop max_tokens clamp (main 2d2bdd04)
Split still planned for reviewability (sampler/serve already split out)
Parity tests for resident/prefill seams still TODO before merge ask

Lab prefill (not blocking this PR thread): SharedK-WMMA + PREFILL_GEMM_M=2048 ≈ 2010 eng @~11k on dual R9700 FP8 MoE; still working toward ~3k. Will rebase #228 onto main after #234/#227 clear conflicts.

bakon11 · 2026-08-10T14:03:26Z

Lab tip 9a009eb6 now includes SharedK-WMMA + GEMM_M=2048 + PEER_ACT default ON (~2057 eng @11k). Body refreshed. Hold merge until #234 then #227 land; then full rebase + docs/pr-size pass. Best recipe handoff on lab host: bash ~/llms/scripts/vllm/start-gemma4-fp8-8010.sh → http://127.0.0.1:8010/v1.

Rebuild mudler#228 onto current main. Drop workarounds that already landed: - max_tokens clamp / -1→4096 (main 2d2bdd0 SERVE-MAXTOKENS-UNSET) - generation_config eos merge (main fb2dfac / mudler#262) Keep operator VT_SERVER_MAX_NEW_TOKENS for *positive* oversized requests only. Non-positive remains UNSET → max_model_len - seq_len. Includes lab wins: SharedK-WMMA, PREFILL_GEMM_M=2048, PEER_ACT default ON, SSE keepalives + deferred add_request for long Hermes prefill, prefill progress tok_s, rocm_sample.hip for V1 sampler ops. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-10T15:51:38Z

Rebased on tip-of-main — clamp + eos dropped

Per maintainer note (2d2bdd04 / row/SERVE-MAXTOKENS-UNSET) and follow-up on eos (fb2dfac7 / #262):

Dropped from this PR	Now owned by main
`protocol.cpp` `-1`→4096 / soft cap	Non-positive `max_tokens` = UNSET → `max_model_len - seq_len`
`serving_chat` clamp on `before < 1`	Only `VT_SERVER_MAX_NEW_TOKENS` for positive oversize (unchanged policy)
`hf_config` / FEATURES multi-EOS merge	generation_config eos via #262

Head: e6c8c4e9 (+ follow-up newline fix) — single rebuild tip on origin/main 848d4a87 (not a multi-commit rebase fight).

Still in this PR: FP8 resident MoE, SharedK-WMMA prefill, PEER_ACT default ON, GEMM_M=2048, SSE keepalives + deferred add_request (Hermes long-prefill), prefill progress tok_s, rocm_sample.hip.

Land order preference unchanged: #234 sampler → #227 KV fail-fast → this #228 (sampler TU also present here so either order can work after a final rebase).

Local: check-device-leakage / check-env-doc / check-public-doc-tables green; HIP server build OK on lab dual-R9700.

Tip CMake had dropped main's DSpark sources and HIP -O1 floor (mudler#132). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-10T15:52:15Z

Follow-up on the rebuild tip: CMakeLists restored from main (DSpark + HIP -O1 floor kept) with only rocm_sample.hip added — HIP server links clean on lab. Tip now 136d9d74.

…ll-reliability

bakon11 · 2026-08-10T23:04:39Z

Status — hold merge; split next (back on track)

Land order in progress:

feat(rocm): V1 sampling ops for Hermes EngineCore #234 — rebuilt tip-of-main single commit a418791e (local gates green; CI re-run)
fix(server): fail-fast unschedulable KV waits #227 — rebuilt tip-of-main clean fail-fast only ac5725fb (no metrics/spec clobber; local gates green; CI re-run)
This feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA prefill (RDNA4) #228 — hold. Will split per maintainer review before next push:
- PR A: SSE/keepalive with timed wait + streaming tests (no 50ms poll)
- PR B: ROCm/gemma4 only — restore HIPBLAS_V2, no unguarded graph flip, HIP-guard MoE policy, WMMA arch guard, spec/issue/tests

Lab decode WIP (BPE heap, structural-tag default-off, FP8 HW cvt, KV-split/slide, etc.) lives on bakon11 wip/lab-decode-20260810 — not in this PR until split/gates.

bakon11 · 2026-08-10T23:13:47Z

Split in flight

PR	Scope
#234	ROCm V1 sampler
#227	KV fail-fast
#316	SSE keepalives (`get_for` timed wait)
new ROCm split	`feat/gemma4-rocm-fp8-split` — FP8 MoE + SharedK-WMMA only

#228 = hold / close when replaced. No merge-as-is.

bakon11 · 2026-08-11T00:00:41Z

Superseded by #317

This megapr head is behind main and carries mixed serve/docs/history noise.

Land path (split):

feat(rocm): V1 sampling ops for Hermes EngineCore #234 — ROCm V1 sampler (includes parallel RandomSample; Hermes temp>0 decode fix)
fix(server): fail-fast unschedulable KV waits #227 — fail-fast unschedulable KV waits
fix(server): SSE keepalives via collector timed wait #316 — SSE keepalives
feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228) #317 — Gemma4/ROCm FP8 + SharedK-WMMA (this PR’s product payload)

Closing #228 in favor of #317 to avoid dual-tracking. Lab decode stack stays on bakon11 WIP branches, not this PR.

ROCm/Gemma4 half of the mudler#228 split. No OpenAI SSE/serving changes (see mudler#316). No V1 sampler registration (see mudler#234). Includes dual-GPU FP8 resident experts, peer mix helpers, SharedK-WMMA prefill, decode KV-split path, fused_ops seam. Review fixes absorbed in this tip: - Restore HIPBLAS_V2 (ROCm 6.x mudler#201/mudler#132) — was clobbered in megapr - Do NOT flip platforms/rocm support_static_graph_mode (stays main) - Host-side #if VT_ROCWMMA_OK around SharedK-WMMA launches (no empty kernel launch on non-gfx1200/1201 slices) Still follow-ups (called out in PR body): HIP guards in models/, MoE policy FreeBytes/FP8_NATIVE defaults, peer weight_id / hscale, tests/spec. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-11T00:00:42Z

Superseded by split #317 (+ #234/#227/#316). See comment.

ROCm/Gemma4 half of the mudler#228 split. No OpenAI SSE/serving changes (see mudler#316). No V1 sampler registration (see mudler#234). Includes dual-GPU FP8 resident experts, peer mix helpers, SharedK-WMMA prefill, decode KV-split path, fused_ops seam. Review fixes absorbed in this tip: - Restore HIPBLAS_V2 (ROCm 6.x mudler#201/mudler#132) — was clobbered in megapr - Do NOT flip platforms/rocm support_static_graph_mode (stays main) - Host-side #if VT_ROCWMMA_OK around SharedK-WMMA launches (no empty kernel launch on non-gfx1200/1201 slices) Still follow-ups (called out in PR body): HIP guards in models/, MoE policy FreeBytes/FP8_NATIVE defaults, peer weight_id / hscale, tests/spec. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

… (22k cudaStreamSynchronize) Trace of the ~4.5 tok/s state (after native fp4 + bf16-on-GPU): GPU kernels are now fast (native fp4 tensor-core MatmulNvfp4Fp4Native 73% of GPU, bf16 gemvx ~19%, fp4-quant 3.4%), but cudaStreamSynchronize is 78.6% of CUDA-API time — 22,115 calls (~2,760/token), the per-GEMM DrainQueue in the per-expert LqGemmNvfp4Fp4/LqGemm loop (each returns a host std::vector<float>). Same host-orchestration wall the GGUF path hit (task mudler#228 device-resident decode, still pending). Remaining N5 levers to close the ~4x: grouped W4A4 MoE (top_k x 3 per-expert launches+syncs/token -> 3), device-resident decode (keep activations on-device, drain once/step -> kills the 22k syncs), decode CUDA-graph. Scoreboard updated. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…eds a new op/use_a16 + stacking) Records the design finding for the remaining ~4x (host-sync-bound, 22k cudaStreamSynchronize): the grouped-MoE lever that would collapse the top_k x 3 per-expert GEMMs+drains/layer into 3 is NOT drop-in — vt::MoeGroupedGemmNvfp4 is W4A16 (bf16 activation, only scale2; the 35B uses it), wrong numerics for our TRUE-W4A4 experts. Options: (a) a new grouped fp4xfp4 (W4A4) op (fp4-quant the shared token activation once, grouped fp4xfp4 over top_k experts), or (b) the valid use_a16 W4A16 grouped mode (numerics change to verify) — both also need the per-expert Nvfp4Weight vector STACKED at load (like qwen3_5 A3 W2). Needs a spike (task mudler#234, converges with the pending GGUF device-resident-decode mudler#228). Docs + spec N5 updated. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…nt alone is eager-slower; graph is the payoff) Completes the mudler#234 scoping with the decisive precedent from the DeepSeek-V4 device-decode campaign (Bricks A-D, state.md/CLAIM-DEEPSEEK-V4-DEVICE-DECODE): ds4 built exactly the device-resident decode (ForwardResidentDecodeGguf, whole step as one async device chain, per-op syncs dropped) and it was TOKEN-IDENTICAL but EAGER-SLOWER (-20%) — the ~1700 small device-kernel launches/step leave the GPU ~45% idle in host-launch gaps. The sync-drop is a PREREQUISITE; the PAYOFF is the decode CUDA-graph (one cudaGraphLaunch). So the remaining Laguna N5 levers 4+5 are ONE campaign (device-resident THEN graph), the reuse target is qwen3_5's Dev/Nvfp4Dev/ResidentNvfp4/device-SwiGLU machinery, and per the ds4 projection a graph reaches ~10-13 tok/s (clears our ~4.5, may still trail vLLM 18.8). It also lifts the GGUF path (mudler#228 = the same lever). Spec/docs updated; this is a spike-first campaign. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

… correctness 2->13 golden match Implements LagunaMoeResidentFp4: the whole decode token's routed experts as ONE async device chain (per-expert fp4-quant -> MatmulNvfp4Fp4 gate/up, MoeSiluMul, fp4-quant -> down into a stacked [Pk,H], then ONE MoeCombine), draining ONCE instead of ~Pk*3 per-GEMM DrainQueue syncs (the 78.6%-of-API-time wall the nsys found). Unified-memory buffers, all ops on q so CUDA orders them; CUDA-only, the CPU/per-expert path stays the byte-exact reference (run-gate unchanged, test_laguna_nvfp4_loader 3/3). VT_LAGUNA_RESIDENT_MOE (default ON, =0 A/B). MEASURED on GB10 (real 67 GiB ckpt, vLLM's exact prompt ids): - SPEED: decode 0.20 s/tok = EAGER-NEUTRAL vs the per-expert path. This empirically confirms the DeepSeek-V4 device-decode precedent for Laguna: the per-op syncs overlap GPU compute, so the wall is GPU-serial-bound, not sync-bound -- removing the syncs alone does not win; the decode CUDA-graph (collapse the launches, pipeline) is the payoff step. Sync-drop is a PREREQUISITE for the graph, now in place. - CORRECTNESS WIN: the golden-token match jumped 2 -> 13 (ours 22345 83 290 350 674 330 5541 966 340 9626 377 15360 81 ... matches vLLM's MARLIN golden EXACTLY for 13 tokens). The device MoeSiluMul/MoeCombine mirror vLLM's fused-MoE structure far more faithfully than the host GateUpSilu+accumulate. So it lands default-ON: strictly better near-tie at no speed cost. Next: the decode CUDA-graph on this now-device-resident MoE (+ the attention/norm glue) is the remaining payoff lever toward vLLM 18.8 (converges with the GGUF mudler#228 and DeepSeek mudler#231). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…e rest is kernel efficiency Derived from the measured lever-1 state: at 0.20 s/tok the GPU is ~87% busy, so GPU compute is ~0.174 s/tok and host gaps only ~0.026. A PERFECT decode graph (removing all host gaps) caps at ~5.9 tok/s — it clears our ~4.5 but is still 3.3x short of vLLM's 18.8. So the graph is NECESSARY BUT NOT SUFFICIENT for parity; the remaining 3.3x is KERNEL EFFICIENCY: our enabled native fp4 MMA runs ~302us per M=1 expert GEMM (tensor-core tiles waste the single decode row) vs vLLM's tuned cutlass sm120a fp4 GEMM (MatmulNvfp4CutlassModel/DirectD) + FUSED norm+quant / silu+quant kernels. True Laguna-NVFP4 parity is TWO campaigns: (A) device-resident forward + decode graph -> ~5.9 tok/s; (B) route experts to cutlass DirectD + adopt the fused ops + an M=1-tuned fp4 GEMV -> the remaining 3.3x. This materially corrects the earlier "graph is the payoff" framing (the graph alone does not reach 18.8) and is critical for planning mudler#234/mudler#228. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…stant Three open PRs each grew their own workaround for the same defect, with three different answers: mudler#227 clamps to 16, mudler#234 and mudler#228 to 4096. All three would conflict with each other, and all three truncate. This fixes it once, in the one place that owns the translation. Hermes and some OpenAI clients send `max_tokens: -1` for "no client-side limit". `SamplingParams::PostInit` rejects anything below 1, which is what made a clamp look necessary — but a constant is the wrong answer, because "unlimited" already has a correct representation here: UNSET. The engine then generates to `max_model_len - seq_len` (`v1/engine/input_processor.cpp:175-180`, a 1:1 mirror of vllm `input_processor.py:317-321`), which is exactly what the client asked for. With a constant, a client asking for unlimited against a 32k model gets 4096 (or 16) tokens back with `finish_reason=length`, indistinguishable from a real limit it set itself. It also hardcodes a policy number into the protocol layer, where `VT_SERVER_MAX_NEW_TOKENS` is already the tunable. Both request paths are covered: `/v1/completions` (`max_tokens`) and `/v1/chat/completions` (`max_completion_tokens` preferred over `max_tokens`, matching chat_completion/serving.py:299). A positive value is honoured unchanged, and a serving-resolved default still wins over an unset request value. RED before, GREEN after, same test and command, on dgx with the new test built against UNPATCHED main: before: test case THREW exception: max_tokens must be at least 1, got -1 29 cases | 28 passed | 1 failed; 175 assertions | 2 failed after: 29 cases | 29 passed | 0 failed; 179 assertions | 0 failed `test_openai_protocol` gains six subcases pinning the semantics: -1 and 0 unset on both paths, a positive value unchanged, and -1 yielding to a serving default when one is supplied. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…ans UNSET Three open PRs each grew a workaround for the same defect with three different answers: mudler#227 clamps to 16, mudler#234 and mudler#228 to 4096. They would conflict with each other and all three truncate. Fixed once, in the layer that owns the translation, so those PRs can drop it and rebase. Hermes and some OpenAI clients send max_tokens=-1 for "no client-side limit"; PostInit rejects <1, which is what made a clamp look necessary. But "unlimited" already has a correct representation: UNSET, which routes to max_model_len - seq_len (input_processor.cpp:175-180, mirroring vllm input_processor.py:317-321). A constant silently truncates exactly the long-context request that asked to be left unlimited. RED before / GREEN after on dgx, the new test built against UNPATCHED main: before: THREW 'max_tokens must be at least 1, got -1'; 28/29 cases, 2 failed after: 29/29 cases, 179/179 assertions Covers both request paths and documents the semantics in docs/USAGE.md. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…config.json `InputProcessor::generation_config_eos_ids_` is named for generation_config.json but was populated entirely from config.json's `eos_token_id`. Nothing in the LLM path read generation_config.json at all (only Parakeet's loader did), so upstream's "secondary eos ids" were, in our port, just a second copy of the primary list. Upstream loads both. ModelConfig.try_get_generation_config (vllm/config/ model.py) reads generation_config.json whenever --generation-config is "auto" (the default) or "vllm", and SamplingParams.update_from_generation_config (vllm/sampling_params.py:645-655) merges its ids into stop_token_ids. The two files routinely disagree: Gemma-4-26B ships config.json eos: [1, 106] generation_config.json eos: [1, 106, 50] so id 50 — a turn-level stop — never terminated a generation for us. The symptom is a chat model that runs past its stop token to the length cap. Found via mudler#228, which hit this on Gemma-4 and fixed it by merging the ids into `cfg.raw["eos_token_id"]`. That lands them at the wrong layer: raw's `eos_token_id` supplies the PRIMARY eos id, which our existing 1:1 port of update_from_generation_config already gates on `ignore_eos`, while these are SECONDARY stop ids. Rewriting raw would make them unconditional, and — since the primary is `generation_config_eos_ids_.front()` on the list path — a re-sorted union could silently displace which id is primary. So this carries the field separately (`HfConfig::generation_config_eos_ids`), leaves `raw` untouched, and unions by append so the primary keeps its position. The existing ignore_eos gates then apply unchanged: the ids reach stop_token_ids only when ignore_eos is false, and all_stop_token_ids (min_tokens masking) regardless — which is exactly sampling_params.py:653. Red first: both new suites fail 2 assertions each against the current implementation and pass with it. A malformed or absent generation_config.json is a silent no-op, mirroring try_get_generation_config returning {} rather than raising. test_hf_config 194/194 (+5 subcases) test_input_processor 64/64 (+2 subcases) FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

fix(parity): read generation_config.json eos ids We only ever read config.json. `generation_config_eos_ids_` was named for generation_config.json but populated from config.json's `eos_token_id`, and nothing in the LLM path opened generation_config.json at all — only Parakeet's loader did. Upstream reads both (try_get_generation_config for the default --generation-config auto, then sampling_params.py:645-655 merges the ids into stop_token_ids), and the two files routinely disagree: Gemma-4-26B ships config.json [1, 106] against generation_config.json [1, 106, 50]. Id 50 never stopped a generation, so chat models ran to the length cap. Found via mudler#228, which hit it on Gemma-4 but fixed it in raw["eos_token_id"] — the field that supplies the PRIMARY eos, which our existing 1:1 port already gates on ignore_eos. These are SECONDARY stop ids. This carries them in their own field, leaves raw untouched, and unions by append so the primary keeps its position. Red first: both suites fail 2 assertions each without the fix. test_qwen3_paged_engine 184/184 test_qwen27_paged_engine 235/235 SACRED test_qwen36_paged_engine 315/315 SACRED test_hf_config 194/194 test_input_processor 64/64 Both SACRED gates matter here: honoring new stop ids could have shortened gate-model generations and drifted the goldens. They did not move. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

@11k

…dK-WMMA (mudler#41) Lands bakon11's work from PR mudler#317, split out of mudler#228. Merged --no-ff so the contributor commits keep their authorship rather than being re-authored to whoever pressed the button. FP8 resident MoE experts and a SharedK-WMMA prefill path for Gemma-4 on RDNA4 ROCm, plus a CPU seam test that pins the portable vt::ExpertGeGLUFp8* symbols link and the recipe env knobs parse inertly. The concern this PR was originally held on is CLOSED BY MEASUREMENT, not by reading the diff. It was held because ~1,600 added lines sat in gemma4.cpp and gemma4_moe.cpp and Gemma-4 is a gate model. bakon11 rebuilt it to drop the CUDA forward restructure entirely, and both arms were then gated on dgx GB10 in one flock window: `gmain` (main) and `g317` (main + this PR) each returned gemma4-E4B STRICT 32/32 token-exact vs the vLLM 0.25.0 golden. gemma4.cpp is byte-identical to main. Its cuda-fat-build red was never its own: the gencode audit failed on seven files, every one of them Marlin, while this branch touches ROCm and gemma4_moe.cpp and no Marlin gencode at all. That was the drift mudler#407 fixed and the branch predated it -- the same stale-base class that also explained mudler#383 and mudler#393 tonight. ACCEPTED WITH A NAMED RESIDUAL, recorded because it is a product decision and not a gate result. This flips four knobs DEFAULT-ON for ROCm: VT_GEMMA4_PREFILL_PEER_ACT, VT_ATTN_PREFILL_FLASH_SHAREDK, VT_ATTN_PREFILL_SHAREDK_WMMA and VT_ROCM_GRAPH. Non-ROCm builds are unaffected -- every new lever is env-gated off or behind moe.enabled / ROCm ExpertGeGLU dispatch -- and each knob is individually reversible by env. But there is NO AMD CI runner, so those defaults rest entirely on the contributor's lab evidence (2x AMD Radeon AI PRO R9700 gfx1201 / RDNA4, ROCm 7.x, gemma-4-26B-A4B-it-fp8; prefill ~2k @11k, short decode ~45-52 t/s). Feature correctness is unprovable on GB10 and honestly so: all nine cached Gemma checkpoints there are dense bf16 with enable_moe_block false, so RunGemma4Moe, EnsureGemma4Fp8* and ExpertGeGLU* are simply unreachable. The GB10 gate proves NON-REGRESSION, never the feature. Gate: scripts/agent-preflight.sh rc=0 on the merged tree; check-agent-record, check-doc-checkpoint, check-public-doc-tables, check-now-current, check-env-doc and check-commit-trailers all green. CI is queue-blocked (mudler#274), so the operator's own gate run is the authority, as AGENTS.md prescribes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

This was referenced Aug 9, 2026

fix(gemma4/rocm): multi-EOS, host expert LRU, safer device expert admit #205

Closed

feat(rocm): V1 sampling ops for Hermes EngineCore #234

Closed

bakon11 mentioned this pull request Aug 10, 2026

fix(server): fail-fast unschedulable KV waits #227

Closed

4 tasks

localai-org-maint-bot requested changes Aug 10, 2026

View reviewed changes

bakon11 changed the title ~~feat(gemma4/rocm): native FP8 resident, fused decode, GPU prefill MoE~~ feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA prefill (RDNA4) Aug 10, 2026

bakon11 force-pushed the feat/gemma4-fp8-prefill-reliability branch from 9a009eb to e6c8c4e Compare August 10, 2026 15:50

fix(sched): repair prefill progress cerr newlines after rebuild

419cb40

FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

fix(cmake): rebuild CMakeLists from main; only add rocm_sample.hip

136d9d7

Tip CMake had dropped main's DSpark sources and HIP -O1 floor (mudler#132). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

Don Mirror added 2 commits August 10, 2026 15:09

Merge remote-tracking branch 'origin/main' into feat/gemma4-fp8-prefi…

e615607

…ll-reliability

Merge origin/main into feat/gemma4-fp8-prefill-reliability

2893f9c

This was referenced Aug 10, 2026

fix(server): SSE keepalives via collector timed wait #316

Closed

feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228) #317

Merged

bakon11 closed this Aug 11, 2026

Navigation Menu

feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA prefill (RDNA4) - #228