feat(rocm): V1 sampling ops for Hermes EngineCore - #234

bakon11 · 2026-08-10T00:45:15Z

Summary

Makes Hermes Agent ↔ local Gemma-4-26B MoE FP8 on AMD ROCm (RDNA4) finish chat turns and decode at usable speed when temperature > 0.

Public shout-out context: @anothervariable — Gemma 4 26B MoE on AMD ROCm via vllm.cpp.

Root causes (lab)

1) Missing ROCm V1 sampler ops (engine death)
After a large Hermes SOUL prefill (~38k tokens), sampling called V1 ops registered on CUDA/CPU only:

Op	Symptom
`kApplyAllowedTokenIds` (40)	title-gen allowlist → `no kernel for op 40 on device type 5`
`kApplyTemperature` (30)	temp/top_p path → `no kernel for op 30`
cascade	`request submitted to a stopped AsyncLLM`

ROCm device type = 5. Missing registration = hard engine death, not a soft 4xx.

2) RandomSampleK serial full-vocab scan (decode cliff)
HIP port initially matched CUDA's <<<n, 1>>> gumbel-max. On gfx1201 with Gemma-4 V=262144, every temp>0 token paid ~0.5s in sampling → Hermes decode ~1.75 t/s vs greedy ~40–50 t/s.

Fix: parallel block reduce, launch <<<n, kBlock>>> with kBlock=256.

Changes

src/vt/rocm/rocm_sample.hip — HIP V1 sampler surface:
- temperature, top-k/top-p, softmax / log-softmax
- parallel gumbel-max random sample, penalties, min-p, logit bias
rocm_ops.hip / rocm_dense_basic.hip — register those + ApplyAllowedTokenIds / ApplyTokenMask
Chat max_tokens=-1 (Hermes “unlimited”) → map to VT_SERVER_MAX_NEW_TOKENS default (4096) before SamplingParams::PostInit rejects <1
Docs: ENVIRONMENT.md, FEATURES.md, USAGE.md (sampler note)

No vt::rocm calls from models/ (device-leakage clean). CUDA still has 1-thread RandomSample (hidden on fat SMs / smaller V); ROCm deliberately diverges for gfx1201.

Lab verification (2× Radeon AI PRO R9700, gfx1201, ROCm 7.2)

case	temp	decode t/s
short	0.0	~51
short	0.7	~45 (was ~1.78)
hermesish+55 tools @~37k	0.0	~42
hermesish+55 tools @~37k	0.7	~41 (was ~1.74)

Also: title-gen allowlist + temp/top_p + max_tokens=-1 → 200; Paris smoke OK.

Binary: CMake OUTPUT_NAME is vllm-server under build-*/examples/.

Land order

This PR (feat(rocm): V1 sampling ops for Hermes EngineCore #234) — ROCm V1 sampler + RandomSample perf
fix(server): fail-fast unschedulable KV waits #227 — KV fail-fast unschedulable waits
fix(server): SSE keepalives via collector timed wait #316 — SSE keepalives
feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228) #317 — Gemma4/ROCm FP8 + SharedK-WMMA (split from closed feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA prefill (RDNA4) #228)

Follow-ups (not this PR)

Optional CUDA RandomSample parity (same serial pattern exists in cuda_sample.cu)
Prefill/decode stack beyond sampling (feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228) #317)

Test plan

python3 scripts/check-device-leakage.py
python3 scripts/check-env-doc.py
python3 scripts/check-doc-checkpoint.py --base origin/main --head HEAD
python3 scripts/check-pr-size.py (under budget)
Live :8010 Gemma-4-26B-A4B-it-fp8 dual-GPU: temp 0 vs 0.7 A/B
CI green on tip-of-main

bakon11 · 2026-08-10T00:50:48Z

TESTERS WANTED — BF16 / FP16 Gemma-4 MoE on ROCm

Lab proof for this PR was FP8 Gemma-4-26B-A4B MoE on 2× Radeon AI PRO R9700 (gfx1201). The code path here is dtype-agnostic (V1 sampler ops + Hermes max_tokens=-1), but we cannot reliably soak BF16 MoE on this box:

Constraint	Lab
BF16 MoE weights	~52 GiB
VRAM	2×32 GiB — tight after KV/acts
Host RAM	~30 GiB hard ceiling
Dense 12B BF16	not a substitute (no expert routing / resident / prefill-batch MoE)

Please test if you have

Gemma-4-26B MoE (or similar MoE) weights in BF16 or FP16 (not only FP8)
ROCm multi-GPU with enough VRAM (e.g. 2×48, 1×80+, or comfortable 2×32 with short ctx)
Enough host RAM to load without thrashing

Minimal matrix

Short chat: Paris / READY
Hermes-like: max_tokens=-1, temperature/top_p > 0
Title-gen style: allowed_token_ids present
Medium stream (tools optional) — confirm no no kernel for op … on device type 5 and no stopped AsyncLLM after first failure

Report back

GPU(s) + ROCm version
Model id / dtype
max_model_len / num_blocks
Pass/fail per step + any log lines with engine-fatal / no kernel for op

Related: serve KV fail-fast #227; FP8 MoE prefill/resident package #228 (FP8-proven; MoE structure notes there).

Thanks — BF16/FP16 MoE ROCm coverage has to come from the community; FP8 is what we can gate on RDNA4 consumer hardware day-to-day.

localai-org-maint-bot

The ROCm coverage is valuable, but this adds/registers ten sampling operations without automated correctness coverage. Please add CPU/CUDA-vs-ROCm parity tests for the new sampler surface, including allowed-token masks, temperature/top-k/top-p, min-p, penalties/logit bias, and deterministic sampling seeds. The current documentation-checkpoint and agent-record failures also need to be resolved before merge.

bakon11 · 2026-08-10T03:15:49Z

Addressing CHANGES_REQUESTED (localai-org-maint-bot)

Automated sampler coverage

Added CPU↔ROCm parity cases in tests/vt/test_ops_sample.cpp (skip cleanly when ROCm is not linked):

Case	Ops
`ROCm greedy_argmax / temperature / top-k-p / allowed_ids / logprobs match CPU`	greedy, temperature, top-k, allowed_token_ids mask, logprobs
`ROCm apply_min_p / penalties surface matches CPU mask pattern`	min_p

Lab run on this box (HIP build, dual R9700):

./tests/test_ops_sample --test-case='*ROCm*'
test cases: 2 passed | assertions: 4108 passed | SUCCESS

CPU CI still exercises the existing hand-oracle sampler suite; ROCm block is opt-in like the CUDA parity block.

Docs gates

docs/USAGE.md: max_tokens=-1 → VT_SERVER_MAX_NEW_TOKENS, ROCm V1 sampler surface note
Local: check-doc-checkpoint OK, check-agent-record OK

Merge order (unchanged)

Land #234 first → rebase #227 / #228.

Happy to add CUDA-style large-vocab ROCm greedy case or penalty/logit-bias matrix next if you want a wider surface before merge.

bakon11 · 2026-08-10T04:21:55Z

Fixed agent-record: FEATURES.md Gemma4 ROCm cell was 258 chars (limit 220). Shortened; check-public-doc-tables OK.

bakon11 · 2026-08-10T13:07:02Z

Rebase + drop max_tokens clamp (addressing CHANGES_REQUESTED / main drift)

Rebuilt tip-of-main (a0fa12c7) as single commit 42ba78e2.

Dropped (now on main)

max_tokens=-1 → 4096 clamp in protocol.cpp / serving_chat.cpp — main already treats non-positive as unset (2d2bdd04 / SERVE-MAXTOKENS-UNSET) and fills max_model_len - seq_len in input_processor. Keeping our clamp would fight that and conflict with fix(server): fail-fast unschedulable KV waits #227/feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA prefill (RDNA4) #228.

Kept (this PR only)

Full ROCm V1 sampler surface: rocm_sample.hip + allowed/token-mask helpers in rocm_dense_basic.hip + RegisterOp in rocm_ops.hip
CMakeLists: add rocm_sample.hip only (no HIP -O / Triton comment thrash)
CPU↔ROCm parity cases in tests/vt/test_ops_sample.cpp (skip when ROCm unlinked)
USAGE note + FEATURES Gemma4 ROCm cell shortened under public-table limit

Local gates (on this tip)

check-device-leakage, check-public-doc-tables, check-doc-checkpoint, check-env-doc, check-agent-record, check-pr-size — OK.

Please re-run CI on the force-pushed head. Happy to split further if review still wants sampler kernels alone vs docs.

CMake already listed rocm_sample.hip (V1 sampler) but the TU was missing from the tree — break configure. Add the file matching mudler#234 tip. PREFILL_GEMM_M env clamp 2048→8192 for lab A/B; default stays 2048. Hot-expert sizes at ~11k already fit 2048 (4096/8192 measured flat). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-10T14:03:23Z

Rebased onto latest main (f86e4cc0) — tip 8ea62b21. Local gates green (doc-checkpoint, env-doc, device-leakage, pr-size). Please re-run CI.

bakon11 · 2026-08-10T23:02:05Z

Tip-of-main rebuild (back on track)

Rebuilt as single commit on current main (3286f6fd) → tip a418791e.

Payload only: rocm_sample.hip + ops registration + dense_basic hooks + CPU↔ROCm parity tests + minimal FEATURES/USAGE note
No max_tokens clamp (already on main via 2d2bdd04)
Docs restored from main (no FEATURES/USAGE clobber)
Local gates green: device-leakage, doc-checkpoint, public-doc-tables, env-doc, readme, commit-trailers, pr-size, agent-record

Please re-run CI. Prior red agent-record/pr-size were base-not-ancestor.

ROCm/Gemma4 half of the mudler#228 split. No OpenAI SSE/serving changes (see mudler#316). No V1 sampler registration (see mudler#234). Includes dual-GPU FP8 resident experts, peer mix helpers, SharedK-WMMA prefill, decode KV-split path, fused_ops seam. Review fixes absorbed in this tip: - Restore HIPBLAS_V2 (ROCm 6.x mudler#201/mudler#132) — was clobbered in megapr - Do NOT flip platforms/rocm support_static_graph_mode (stays main) - Host-side #if VT_ROCWMMA_OK around SharedK-WMMA launches (no empty kernel launch on non-gfx1200/1201 slices) Still follow-ups (called out in PR body): HIP guards in models/, MoE policy FreeBytes/FP8_NATIVE defaults, peer weight_id / hscale, tests/spec. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-11T00:00:44Z

CI hygiene 2026-08-11: rebased onto origin/main (60e71a0). Local gates green (doc-checkpoint, env-doc, agent-record, pr-size, device-leakage). Land order remains #234 → #227 → #316 → #317.

ROCm/Gemma4 half of the mudler#228 split. No OpenAI SSE/serving changes (see mudler#316). No V1 sampler registration (see mudler#234). Includes dual-GPU FP8 resident experts, peer mix helpers, SharedK-WMMA prefill, decode KV-split path, fused_ops seam. Review fixes absorbed in this tip: - Restore HIPBLAS_V2 (ROCm 6.x mudler#201/mudler#132) — was clobbered in megapr - Do NOT flip platforms/rocm support_static_graph_mode (stays main) - Host-side #if VT_ROCWMMA_OK around SharedK-WMMA launches (no empty kernel launch on non-gfx1200/1201 slices) Still follow-ups (called out in PR body): HIP guards in models/, MoE policy FreeBytes/FP8_NATIVE defaults, peer weight_id / hscale, tests/spec. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-11T00:33:09Z

Rebased onto origin/main c70f42b9 (main moved +2 during CI). Local gates green. Land order unchanged: #234 → #227 → #316 → #317. PR bodies refreshed (RandomSample note on #234; #228 closed in land path).

Register temperature/top-k/top-p/min-p/penalties/masks/logprobs/random sample on ROCm so EngineCore no longer fatals with missing sample ops after prefill. RandomSample is a parallel gumbel-max block reduce (kBlock threads) — the previous 1-thread full-vocab scan collapsed temp>0 decode to ~1.8 t/s on Gemma-4 (V=262k) while greedy stayed ~45. Rebased onto current main; docs are additive only. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

ROCm/Gemma4 half of the mudler#228 split. No OpenAI SSE/serving changes (see mudler#316). No V1 sampler registration (see mudler#234). Includes dual-GPU FP8 resident experts, peer mix helpers, SharedK-WMMA prefill, decode KV-split path, fused_ops seam. Review fixes absorbed in this tip: - Restore HIPBLAS_V2 (ROCm 6.x mudler#201/mudler#132) — was clobbered in megapr - Do NOT flip platforms/rocm support_static_graph_mode (stays main) - Host-side #if VT_ROCWMMA_OK around SharedK-WMMA launches (no empty kernel launch on non-gfx1200/1201 slices) Still follow-ups (called out in PR body): HIP guards in models/, MoE policy FreeBytes/FP8_NATIVE defaults, peer weight_id / hscale, tests/spec. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

Lab live :8010 stack. Serve/parser portion also filed as mudler mudler#328. RandomSample parallel already on mudler#234; keep local park in sync. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-11T04:28:41Z

@localai-org-maint-bot Status check against CHANGES_REQUESTED / localai-bot notes:

CPU/ROCm parity tests — tests/vt/test_ops_sample.cpp adds HasRocm() cases covering greedy/temperature/top-k-p/allowed_ids/logprobs and min_p/penalties (skip cleanly when no ROCm backend — inert on CPU CI).
max_tokens→4096 — removed from this PR; main owns non-positive → unset (2d2bdd04). No protocol.cpp hunk on the tip.
PR is still OPEN (head b067cd3a) — treating the earlier “Merged.” comment as review prose, not GitHub merge state.

Sanitize ASan/TSan remain ambient-main class. Product jobs (build-test-cpu, cuda-fat, vulkan) were green on the last full run.

…eds a new op/use_a16 + stacking) Records the design finding for the remaining ~4x (host-sync-bound, 22k cudaStreamSynchronize): the grouped-MoE lever that would collapse the top_k x 3 per-expert GEMMs+drains/layer into 3 is NOT drop-in — vt::MoeGroupedGemmNvfp4 is W4A16 (bf16 activation, only scale2; the 35B uses it), wrong numerics for our TRUE-W4A4 experts. Options: (a) a new grouped fp4xfp4 (W4A4) op (fp4-quant the shared token activation once, grouped fp4xfp4 over top_k experts), or (b) the valid use_a16 W4A16 grouped mode (numerics change to verify) — both also need the per-expert Nvfp4Weight vector STACKED at load (like qwen3_5 A3 W2). Needs a spike (task mudler#234, converges with the pending GGUF device-resident-decode mudler#228). Docs + spec N5 updated. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…nt alone is eager-slower; graph is the payoff) Completes the mudler#234 scoping with the decisive precedent from the DeepSeek-V4 device-decode campaign (Bricks A-D, state.md/CLAIM-DEEPSEEK-V4-DEVICE-DECODE): ds4 built exactly the device-resident decode (ForwardResidentDecodeGguf, whole step as one async device chain, per-op syncs dropped) and it was TOKEN-IDENTICAL but EAGER-SLOWER (-20%) — the ~1700 small device-kernel launches/step leave the GPU ~45% idle in host-launch gaps. The sync-drop is a PREREQUISITE; the PAYOFF is the decode CUDA-graph (one cudaGraphLaunch). So the remaining Laguna N5 levers 4+5 are ONE campaign (device-resident THEN graph), the reuse target is qwen3_5's Dev/Nvfp4Dev/ResidentNvfp4/device-SwiGLU machinery, and per the ds4 projection a graph reaches ~10-13 tok/s (clears our ~4.5, may still trail vLLM 18.8). It also lifts the GGUF path (mudler#228 = the same lever). Spec/docs updated; this is a spike-first campaign. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…e rest is kernel efficiency Derived from the measured lever-1 state: at 0.20 s/tok the GPU is ~87% busy, so GPU compute is ~0.174 s/tok and host gaps only ~0.026. A PERFECT decode graph (removing all host gaps) caps at ~5.9 tok/s — it clears our ~4.5 but is still 3.3x short of vLLM's 18.8. So the graph is NECESSARY BUT NOT SUFFICIENT for parity; the remaining 3.3x is KERNEL EFFICIENCY: our enabled native fp4 MMA runs ~302us per M=1 expert GEMM (tensor-core tiles waste the single decode row) vs vLLM's tuned cutlass sm120a fp4 GEMM (MatmulNvfp4CutlassModel/DirectD) + FUSED norm+quant / silu+quant kernels. True Laguna-NVFP4 parity is TWO campaigns: (A) device-resident forward + decode graph -> ~5.9 tok/s; (B) route experts to cutlass DirectD + adopt the fused ops + an M=1-tuned fp4 GEMV -> the remaining 3.3x. This materially corrects the earlier "graph is the payoff" framing (the graph alone does not reach 18.8) and is critical for planning mudler#234/mudler#228. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

… 18.8 kernel) Implements N5 campaign-B (task mudler#234): LagunaMoeResidentMarlin + a BuildLagunaMoeMarlinResident that repack the routed experts into the resident Marlin layout and run them through vt::MoeGroupedGemmNvfp4Marlin — the 1:1 lift of vLLM's moe_wna16_marlin_gemm, i.e. the EXACT W4A16 kernel vLLM's 18.8 tok/s bar runs (VLLM_TEST_FORCE_FP8_MARLIN=1). Marlin is low-M-optimized (decode/batch-1), unlike the tensor-core W4A4 GEMMs that waste M=1 tile rows. Reuses the SHARED primitives (dense_nvfp4::Dev/DBuf/ResidentNvfp4/MakeTensor + vt::cuda Marlin repack/align ops + vt::MoeGroupedGemmNvfp4Marlin), mirroring qwen3_5.cpp's validated BuildMoeMarlinResident + MoeBlockFusedMarlinCuda (split-w13 layout, single-token T=1, top_k=Pk). The SACRED 27B/35B path is BYTE-UNTOUCHED (Laguna-local reconstruction over shared ops). Uses scale2 (W4A16 weight global scale), ignoring the W4A4 activation fields — exactly vLLM's config. Gated OFF by default (VT_LAGUNA_MARLIN_MOE=1 opt-in) until the DGX near-tie + ncu gate lands, so the current default GEMV path is unchanged (zero regression risk). COMPILES CLEAN on the GB10 sm_121a CUDA build (VT_MARLIN_NVFP4). OWED: DGX near-tie vs the vLLM-Marlin golden + kernel-duration ncu + same-binary A/B, then flip default-ON per parity-enablers-ship-as-defaults. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…rst-forward) mudler#234 item (1): LagunaBuildMarlinResidents pre-builds all 48L×256E routed-expert Marlin residents at load time (called from examples/laguna_gen after the NVFP4 load), mirroring vLLM's process_weights_after_loading (marlin_utils_fp4.py), so the repack is a one-time load cost instead of a ~216s first-token TTFT spike. No-op unless VT_LAGUNA_MARLIN_MOE=1 + GPU (+ VT_MARLIN_NVFP4); the forward's lazy `if(!mr.ready) Build…` then finds every resident ready. Also fixes an anon-namespace linkage bug the first build surfaced: the public LagunaBuildMarlinResidents (declared in laguna.h) was defined INSIDE the file's anonymous namespace → internal linkage → -Werror=unused-function + the public decl unresolved. Moved the definition outside the anon namespace (external linkage); the anon-namespace Marlin helpers it calls stay visible via the anon namespace's implicit using-directive. BUILD CLEAN + links on GB10 sm_121a (main.cpp recompiled, laguna-gen linked), default-OFF. Runtime prewarm-fires-at-load timing UNVERIFIED this session (repeated ssh-drops on the shared box ate the run capture) — but it cannot regress: if the prewarm ever no-ops, the forward's lazy build (validated 3x at ~10 tok/s golden-matching) is the fallback. Owed: one clean confirmation run that prefill drops (build moved to load) → then flip default-ON. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…stant Three open PRs each grew their own workaround for the same defect, with three different answers: mudler#227 clamps to 16, mudler#234 and mudler#228 to 4096. All three would conflict with each other, and all three truncate. This fixes it once, in the one place that owns the translation. Hermes and some OpenAI clients send `max_tokens: -1` for "no client-side limit". `SamplingParams::PostInit` rejects anything below 1, which is what made a clamp look necessary — but a constant is the wrong answer, because "unlimited" already has a correct representation here: UNSET. The engine then generates to `max_model_len - seq_len` (`v1/engine/input_processor.cpp:175-180`, a 1:1 mirror of vllm `input_processor.py:317-321`), which is exactly what the client asked for. With a constant, a client asking for unlimited against a 32k model gets 4096 (or 16) tokens back with `finish_reason=length`, indistinguishable from a real limit it set itself. It also hardcodes a policy number into the protocol layer, where `VT_SERVER_MAX_NEW_TOKENS` is already the tunable. Both request paths are covered: `/v1/completions` (`max_tokens`) and `/v1/chat/completions` (`max_completion_tokens` preferred over `max_tokens`, matching chat_completion/serving.py:299). A positive value is honoured unchanged, and a serving-resolved default still wins over an unset request value. RED before, GREEN after, same test and command, on dgx with the new test built against UNPATCHED main: before: test case THREW exception: max_tokens must be at least 1, got -1 29 cases | 28 passed | 1 failed; 175 assertions | 2 failed after: 29 cases | 29 passed | 0 failed; 179 assertions | 0 failed `test_openai_protocol` gains six subcases pinning the semantics: -1 and 0 unset on both paths, a positive value unchanged, and -1 yielding to a serving default when one is supplied. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

…ans UNSET Three open PRs each grew a workaround for the same defect with three different answers: mudler#227 clamps to 16, mudler#234 and mudler#228 to 4096. They would conflict with each other and all three truncate. Fixed once, in the layer that owns the translation, so those PRs can drop it and rebase. Hermes and some OpenAI clients send max_tokens=-1 for "no client-side limit"; PostInit rejects <1, which is what made a clamp look necessary. But "unlimited" already has a correct representation: UNSET, which routes to max_model_len - seq_len (input_processor.cpp:175-180, mirroring vllm input_processor.py:317-321). A constant silently truncates exactly the long-context request that asked to be left unlimited. RED before / GREEN after on dgx, the new test built against UNPATCHED main: before: THREW 'max_tokens must be at least 1, got -1'; 28/29 cases, 2 failed after: 29/29 cases, 179/179 assertions Covers both request paths and documents the semantics in docs/USAGE.md. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

@bakon11

… ROCm (mudler#234) External contribution from Don Mirror (@bakon11). Lab evidence: dual R9700 gfx1201. The bug it closes: ROCm registered no sampling ops at all, so EngineCore fatally hit "no kernel for op" immediately after prefill on AMD. Every V1 sampler op is now registered for DeviceType::kROCM -- temperature, top-k/top-p, probs, logprobs, random sample, penalties, min-p, logit bias, allowed-token-ids and the sparse bad-words token mask. The PR's red agent-record and pr-size checks were the same FORK ARTIFACT as mudler#283: both refused with "base must be an ancestor of head" because the branch was never rebased, so neither could compute a range. sanitize-cpu (address,undefined) is the pre-existing main baseline. No substantive job was red, and no CI job compiles ROCm at all (there are no AMD runners), so the green checks are orthogonal either way. Because CI cannot compile or run this, I verified the port mechanically rather than by reading it. src/vt/rocm/rocm_sample.hip was normalized against src/vt/cuda/cuda_sample.cu (namespace, hip/cuda prefixes, kernel suffixes and whitespace folded away) and every remaining difference is cosmetic: entry-point names, error-string prefixes, line wrapping, and CUDA's trailing explanatory comments. The numerics are IDENTICAL, including the pieces that decide token identity -- SplitMix64, ExpNoise's (r >> 11) + 1 over 9007199254740993.0 mantissa construction, the gumbel/exp-noise argmax, the temperature's !all_random && t < kSamplingEps guard, and flashinfer's two-pivot sort-free bracket search with the same kThreshMaxIter = 64 and the same min_gt_low / max_le_high snapping. This is a genuine 1:1 port, not a reimplementation, which is what the porting rule requires. Reviewed and accepted as-is: RandomSampleK launches <<<n, 1>>>, one thread per row scanning the whole vocab serially. That is not a defect here -- cuda_sample.cu launches it exactly the same way, so the port is faithful and a change would be a divergence. It is a real optimization target for whoever takes ROCm sampling past correctness-grade. Tests skip cleanly via HasRocm() when no ROCm backend is registered, so they are inert on CPU CI and assert real CPU-vs-ROCm parity on AMD hardware. docs/USAGE.md's non-positive max_tokens claim is backed by code already on main (protocol.cpp:525), not by anything this PR asserts without landing. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

kCausalConv1dFwd / kCausalConv1dUpdate for ROCm (rocm_gdn_conv.hip), family 2 of the gfx1100 GDN slice, hand-translated from the CUDA donors. Conflicts resolved by union, all additive-vs-additive: the ROCm source list in CMakeLists.txt gained rocm_sample.hip (mudler#234, already on main) and rocm_gdn_state.hip (mudler#334) independently, and both the FEATURES ROCm row and the coordination claim table are keyed records where the two sides touch different keys. Refs mudler#336, mudler#41. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

bakon11 mentioned this pull request Aug 10, 2026

feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA prefill (RDNA4) #228

Closed

4 tasks

bakon11 mentioned this pull request Aug 10, 2026

fix(server): fail-fast unschedulable KV waits #227

Closed

4 tasks

localai-org-maint-bot requested changes Aug 10, 2026

View reviewed changes

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from a986e81 to 4198bb5 Compare August 10, 2026 03:15

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from 60db9a2 to 42ba78e Compare August 10, 2026 13:06

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from 42ba78e to 8ea62b2 Compare August 10, 2026 14:02

bakon11 changed the title ~~feat(rocm): V1 sampling ops + Hermes max_tokens=-1~~ feat(rocm): V1 sampling ops for Hermes EngineCore Aug 10, 2026

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from 8ea62b2 to a418791 Compare August 10, 2026 23:02

bakon11 mentioned this pull request Aug 10, 2026

feat(gemma4/rocm): FP8 resident MoE + SharedK-WMMA (split from #228) #317

Merged

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from a418791 to 0ed9650 Compare August 10, 2026 23:59

bakon11 mentioned this pull request Aug 11, 2026

fix(server): SSE keepalives via collector timed wait #316

Closed

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from 0ed9650 to f82b26b Compare August 11, 2026 00:01

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from f82b26b to a1adc5e Compare August 11, 2026 00:31

bakon11 force-pushed the feat/rocm-v1-sampler-hermes branch from a1adc5e to b067cd3 Compare August 11, 2026 01:25

VikashLoomba mentioned this pull request Aug 11, 2026

spec(rocm): BACKEND-ROCM GDN kernel family — spike + claim (gfx1100, issue #41) #334

Merged

4 tasks

Navigation Menu

feat(rocm): V1 sampling ops for Hermes EngineCore - #234