Skip to content

feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228) - #317

Merged
localai-bot merged 2 commits into
mudler:mainfrom
bakon11:feat/gemma4-rocm-fp8-split
Aug 11, 2026
Merged

feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228)#317
localai-bot merged 2 commits into
mudler:mainfrom
bakon11:feat/gemma4-rocm-fp8-split

Conversation

@bakon11

@bakon11 bakon11 commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

ROCm / Gemma4 half of the #228 split. #228 is closed — land this instead.

Related Role
#234 ROCm V1 sampler + parallel RandomSample
#227 KV fail-fast
#316 SSE keepalives (timed wait) — not here
#228 Original megapr — closed, superseded

In

Dual-GPU FP8 resident MoE, peer mix, SharedK-WMMA prefill, decode KV-split, fused_ops seam, ENVIRONMENT VT_GEMMA4_* / VT_ATTN_*.

Out

SSE/serving (#316), rocm_sample (#234), support_static_graph_mode flip.

Review fixes in this tip

  • HIPBLAS_V2 restored
  • No platform graph-mode flip
  • Host #if VT_ROCWMMA_OK around SharedK-WMMA launches

Honest follow-ups

HIP guards in models/gemma4.cpp, MoE policy guards, peer weight_id/hscale, tests/spec. Large diff (~5k prod lines) — may need another slice or maintainer path like #154 if review asks.

Lab: dual R9700 gfx1201 / start-gemma4-fp8-8010.sh.

Land order

  1. feat(rocm): V1 sampling ops for Hermes EngineCore #234 → 2. fix(server): fail-fast unschedulable KV waits #227 → 3. fix(server): SSE keepalives via collector timed wait #316 → 4. This PR (feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228) #317)

@bakon11
bakon11 force-pushed the feat/gemma4-rocm-fp8-split branch from b73018c to 72e734f Compare August 11, 2026 00:00
This was referenced Aug 11, 2026
Closed
Closed
Closed
@bakon11

bakon11 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

CI hygiene 2026-08-11: rebased onto origin/main (60e71a0). Local gates green (doc-checkpoint, env-doc, agent-record, pr-size, device-leakage). Land order remains #234 → #227 → #316 → #317.

@bakon11 bakon11 mentioned this pull request Aug 11, 2026
Closed
@bakon11
bakon11 force-pushed the feat/gemma4-rocm-fp8-split branch 2 times, most recently from b7a0622 to 2f58e58 Compare August 11, 2026 00:31
@bakon11

bakon11 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main c70f42b9 (main moved +2 during CI). Local gates green. Land order unchanged: #234 → #227 → #316 → #317. PR bodies refreshed (RandomSample note on #234; #228 closed in land path).

@bakon11 bakon11 mentioned this pull request Aug 11, 2026
Closed
3 tasks
@bakon11
bakon11 force-pushed the feat/gemma4-rocm-fp8-split branch from 2f58e58 to 606e894 Compare August 11, 2026 01:25
@VikashLoomba VikashLoomba mentioned this pull request Aug 11, 2026
Merged
4 tasks
localai-org-maint-bot
localai-org-maint-bot requested changes Aug 11, 2026
View reviewed changes

@localai-org-maint-bot localai-org-maint-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bakon11 This is not reviewable/mergeable with the PR body itself listing tests/spec as honest follow-ups. The ~5k-line production change needs the project-required spike/spec, focused automated parity/regression coverage for the new resident MoE/peer/SharedK/decode paths, and the same-change STATUS/BENCHMARKS/matrix/state evidence anchors required by AGENTS.md. Please land those in this PR (or split the stack further) before asking for merge.

@bakon11

bakon11 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

@localai-org-maint-bot Addressed the mergeability blockers you named:

  1. Spike/spec landed: .agents/specs/gemma4-rocm-fp8-moe.md (scope, gates, residuals, merge criteria).
  2. Focused automated coverage: new test_gemma4_rocm_fp8_seams (CPU) — portable vt::ExpertGeGLUFp8* / FP8 channel symbols link, recipe env knobs parse inertly. No AMD CI runners exist; ROCm kernel numerical parity remains lab evidence (dual R9700) as documented in the spec.
  3. FEATURES row now anchors the spec + seam test (AGENTS public-doc surface).
  4. PR body no longer lists tests/spec as vague follow-ups — residuals are named.

Local: test_gemma4_rocm_fp8_seams green. Device-leakage / env-doc / trailer checks OK on tip.

bakon11 pushed a commit to bakon11/vllm.cpp that referenced this pull request Aug 11, 2026
Don Mirror
docs(test): Gemma4 ROCm FP8 MoE spike/spec + CPU seam gate
17df82f 
Maint-bot mudler#317: land the project-required spike/spec and a focused automated
CPU gate for the portable fused_ops FP8 ExpertGeGLU seams (no AMD runner).
Lab RDNA4 quality/speed remains recipe evidence; CUDA SACRED gates unchanged.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Hermes:grok-4.5 [Hermes]
bakon11 pushed a commit to bakon11/vllm.cpp that referenced this pull request Aug 11, 2026
Don Mirror
feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (no CUDA forward r…
a5bd7ad 
…estructure)

Rebuild of mudler#317 on current origin/main. Keeps ROCm FP8 ExpertGeGLU, SharedK-WMMA
prefill, fused_ops seams, expert LRU/prewarm, and CPU seam test.

Deferred (localai-bot hold): ForwardGemma4Layers extract, Gemma4DecodeGraph, and
any unguarded gemma4.cpp layer-loop restructure that would require a GB10
token-exact golden. Those remain lab-only until a separate CUDA-gated PR.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Hermes:grok-4.5 [Hermes]
@bakon11
bakon11 force-pushed the feat/gemma4-rocm-fp8-split branch from 17df82f to a5bd7ad Compare August 11, 2026 12:05
@bakon11

bakon11 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebuilt on current main — addresses hold (CUDA forward restructure)

Tip: a5bd7ad0 on origin/main c43092f9 (ancestor YES, behind=0).

What changed vs prior tip

Prior hold called out unguarded gemma4.cpp / gemma4_moe.cpp forward work and ForwardGemma4Layers + layer TLS + Gemma4DecodeGraph (CUDA path / graph lifetime).

This rebuild drops all of that from the PR:

Deferred (lab-only / later PR) Still in #317
ForwardGemma4Layers extract ROCm FP8 ExpertGeGLU + channel GEMV
Gemma4DecodeGraph / decode hipGraph SharedK + SharedK-WMMA prefill
MLP∥MoE side-stream wiring vt::fused_ops portable seams
Any CUDA-path layer-loop rewrite Expert LRU / host cache / prewarm
GB10 token-exact requirement for this tip DeviceMemoryInfo backend seam
CPU seam test test_gemma4_rocm_fp8_seams
Spec + ENVIRONMENT/FEATURES/USAGE

gemma4.cpp / gemma4.h are unchanged vs main in this tip. Registry only adds ExpertGeGLU prewarm after resident upload (no graph hook).

Bot checklist

  1. Token-exact GB10 — N/A for this tip (no forward restructure). Happy to land graph/extract later with golden.
  2. TLS vs graph capture — N/A (no Gemma4 decode graph in PR).
  3. thread_local lifetime — only existing main layer TLS remains; no new capture driver.

Local gates (green)

Sanitize-cpu ASan/TSan ambient noise may still red; not specific to this payload.

Please re-review when CI settles on a5bd7ad0.

@bakon11
bakon11 force-pushed the feat/gemma4-rocm-fp8-split branch from a5bd7ad to d1c4a65 Compare August 11, 2026 15:11
bakon11 pushed a commit to bakon11/vllm.cpp that referenced this pull request Aug 11, 2026
Don Mirror
feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (no CUDA forward r…
d1c4a65 
…estructure)

Rebuild of mudler#317 on current origin/main. Keeps ROCm FP8 ExpertGeGLU, SharedK-WMMA
prefill, fused_ops seams, expert LRU/prewarm, and CPU seam test.

Deferred (localai-bot hold): ForwardGemma4Layers extract, Gemma4DecodeGraph, and
any unguarded gemma4.cpp layer-loop restructure that would require a GB10
token-exact golden. Those remain lab-only until a separate CUDA-gated PR.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Hermes:grok-4.5 [Hermes]
@bakon11

bakon11 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current main (post-#377 gate fix)

Tip: d1c4a656 on current origin/main (clean rebase of prior a5bd7ad0; behind=0).

Thanks for the GB10 hardware check — confirmed:

  1. CUDA forward concern closed (no ForwardGemma4Layers / no new shape-keyed TLS; gemma4.cpp untouched vs main).
  2. SACRED red was main’s GeluMulSeparate (Gemma-4 text generation throws on every CUDA build: shared forward calls the ROCm-only vt::GeluMulSeparate (SACRED gate RED on main since 0c2827c1) #377) — not this PR. Rebased so the gate can run on the fixed baseline.

What CUDA / non-ROCm builds do with this tip

  • gemma4.cpp / gemma4.h: identical to main → no CUDA path change.
  • gemma4_moe.cpp: shared file, but every new lever is env-gated default-off or behind moe.enabled / ROCm ExpertGeGLU dispatch. On CUDA/CPU/Metal/Vulkan without the ROCm FP8 MoE path: no new default behavior; resident FP8 / ExpertGeGLU / peer-act do not activate.
  • GB10 dense bf16 Gemma (enable_moe_block: false): MoE/FP8 ExpertGeGLU / registry prewarm remain unreachable — expected. Feature validation is ROCm + MoE FP8 ckpt only.

ROCm lab evidence (this box — not GB10)
Board 2× AMD Radeon AI PRO R9700 (gfx1201 / RDNA4), ROCm 7.x
Checkpoint gemma-4-26B-A4B-it-fp8 (HF FP8 MoE)
Serve ~/llms/scripts/vllm/start-gemma4-fp8-8010.sh → :8010
Measured class Prefill eng ~2k @11k (SharedK-WMMA + PEER_ACT + GEMM_M=2048); short decode ~45–52 t/s; long pad quality Paris OK under BF16 KV default

Default-on ROCm knobs (explicit)

These change ROCm default behavior when unset (documented in ENVIRONMENT); non-ROCm unaffected:

  • VT_GEMMA4_PREFILL_PEER_ACT default on
  • VT_ATTN_PREFILL_FLASH_SHAREDK default on
  • VT_ATTN_PREFILL_SHAREDK_WMMA default on
  • VT_ROCM_GRAPH default on (no Gemma4 decode-graph driver in this PR)

KV cache dtype (separate from weight FP8)

Serve selection is env today (llama.cpp-style intent, different surface):

  • default bf16 KV with every weight quant
  • VT_KV_CACHE_DTYPE=fp8|bf16|… or VT_KV_CACHE_FP8=1 opt-in
    CLI --kv-cache-dtype twin not in this PR (follow-up).

Local gates on d1c4a656

device-leakage / env-doc / doc-checkpoint / agent-record / pr-size / commit-trailers — green.

Please re-run CI + SACRED on the new tip when ready.

Don Mirror added 2 commits August 11, 2026 17:15
Don Mirror
feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (no CUDA forward r…
9302732 
…estructure)

Rebuild of mudler#317 on current origin/main. Keeps ROCm FP8 ExpertGeGLU, SharedK-WMMA
prefill, fused_ops seams, expert LRU/prewarm, and CPU seam test.

Deferred (localai-bot hold): ForwardGemma4Layers extract, Gemma4DecodeGraph, and
any unguarded gemma4.cpp layer-loop restructure that would require a GB10
token-exact golden. Those remain lab-only until a separate CUDA-gated PR.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Hermes:grok-4.5 [Hermes]
Don Mirror
docs(features): Gemma4 ROCm FP8 row inside 220-char cell budget
c03304d 
FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Hermes:grok-4.5 [Hermes]
@bakon11
bakon11 force-pushed the feat/gemma4-rocm-fp8-split branch from 984f0c1 to c03304d Compare August 11, 2026 22:15
@localai-bot
localai-bot merged commit 437059c into mudler:main Aug 11, 2026
14 checks passed
@VikashLoomba VikashLoomba mentioned this pull request Aug 14, 2026
Open
6 tasks
Frenchy2k1 pushed a commit to Frenchy2k1/vllm.cpp_sm_70 that referenced this pull request Aug 29, 2026
Don Mirror
feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (no CUDA forward r…
2a48d69 
…estructure)

Rebuild of mudler#317 on current origin/main. Keeps ROCm FP8 ExpertGeGLU, SharedK-WMMA
prefill, fused_ops seams, expert LRU/prewarm, and CPU seam test.

Deferred (localai-bot hold): ForwardGemma4Layers extract, Gemma4DecodeGraph, and
any unguarded gemma4.cpp layer-loop restructure that would require a GB10
token-exact golden. Those remain lab-only until a separate CUDA-gated PR.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:deepseek-v4-flash [edit bash]
Frenchy2k1 pushed a commit to Frenchy2k1/vllm.cpp_sm_70 that referenced this pull request Aug 29, 2026
@mudler
merge: land row/GEMMA4-ROCM-FP8-SPLIT -- Gemma-4 ROCm FP8 MoE + Share…
081b795 
…dK-WMMA (mudler#41)

Lands bakon11's work from PR mudler#317, split out of mudler#228. Merged --no-ff so the
contributor commits keep their authorship rather than being re-authored to
whoever pressed the button.

FP8 resident MoE experts and a SharedK-WMMA prefill path for Gemma-4 on RDNA4
ROCm, plus a CPU seam test that pins the portable vt::ExpertGeGLUFp8* symbols
link and the recipe env knobs parse inertly.

The concern this PR was originally held on is CLOSED BY MEASUREMENT, not by
reading the diff. It was held because ~1,600 added lines sat in gemma4.cpp and
gemma4_moe.cpp and Gemma-4 is a gate model. bakon11 rebuilt it to drop the CUDA
forward restructure entirely, and both arms were then gated on dgx GB10 in one
flock window: `gmain` (main) and `g317` (main + this PR) each returned
gemma4-E4B STRICT 32/32 token-exact vs the vLLM 0.25.0 golden. gemma4.cpp is
byte-identical to main.

Its cuda-fat-build red was never its own: the gencode audit failed on seven
files, every one of them Marlin, while this branch touches ROCm and
gemma4_moe.cpp and no Marlin gencode at all. That was the drift mudler#407 fixed and
the branch predated it -- the same stale-base class that also explained mudler#383 and
mudler#393 tonight.

ACCEPTED WITH A NAMED RESIDUAL, recorded because it is a product decision and
not a gate result. This flips four knobs DEFAULT-ON for ROCm:
VT_GEMMA4_PREFILL_PEER_ACT, VT_ATTN_PREFILL_FLASH_SHAREDK,
VT_ATTN_PREFILL_SHAREDK_WMMA and VT_ROCM_GRAPH. Non-ROCm builds are unaffected --
every new lever is env-gated off or behind moe.enabled / ROCm ExpertGeGLU
dispatch -- and each knob is individually reversible by env. But there is NO AMD
CI runner, so those defaults rest entirely on the contributor's lab evidence
(2x AMD Radeon AI PRO R9700 gfx1201 / RDNA4, ROCm 7.x, gemma-4-26B-A4B-it-fp8;
prefill ~2k @11k, short decode ~45-52 t/s). Feature correctness is unprovable on
GB10 and honestly so: all nine cached Gemma checkpoints there are dense bf16 with
enable_moe_block false, so RunGemma4Moe, EnsureGemma4Fp8* and ExpertGeGLU* are
simply unreachable. The GB10 gate proves NON-REGRESSION, never the feature.

Gate: scripts/agent-preflight.sh rc=0 on the merged tree; check-agent-record,
check-doc-checkpoint, check-public-doc-tables, check-now-current, check-env-doc
and check-commit-trailers all green. CI is queue-blocked (mudler#274), so the
operator's own gate run is the authority, as AGENTS.md prescribes.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:deepseek-v4-flash [edit bash]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@localai-org-maint-bot localai-org-maint-bot localai-org-maint-bot requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bakon11 @localai-org-maint-bot