fix(server): gemma4 bare call: tool extract for OpenAI chat - #328

bakon11 · 2026-08-11T01:25:16Z

Summary

Gemma-4 free-form tool use (no structural tags / specials stripped at detokenize) often emits bare call:NAME{ARGS} instead of wrapped <|tool_call>… bodies. The engine-backed gemma4 parser needs special-token IDs; OpenAI /v1/chat/completions then returned plain content and clients (e.g. Hermes) never executed tools.

Changes

Text-seam Gemma4ToolParser: extract bare call:NAME{ARGS} (stream + non-stream), still supports wrapped blocks
OpenAI chat: for --tool-call-parser gemma4, prefer text-seam MakeToolParser over engine path so detokenized/free-form bodies become real tool_calls
USAGE: document the gemma4 dual surface

Test plan

Lab: non-stream + stream tools=[{terminal}] → finish_reason=tool_calls, args {"command":"date"}
CI gates
Regression: wrapped <|tool_call>call:… still extracts

Notes

Small serve/parser fix; independent of ROCm #317. Does not change structural-tag defaults.

FOLLOWING_AGENTS_PROTOCOL

Free-form Gemma-4 (structural tags off / detokenized specials) often emits bare `call:NAME{ARGS}` instead of wrapped <|tool_call>… bodies. The engine parser needs special-token IDs; OpenAI chat now prefers the text-seam gemma4 tool parser and accepts both wrapped and bare call surfaces so clients get real tool_calls instead of plain content. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

Lab live :8010 stack. Serve/parser portion also filed as mudler mudler#328. RandomSample parallel already on mudler#234; keep local park in sync. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

localai-org-maint-bot

@bakon11 Please add automated stream and non-stream regression coverage before this parser/dispatch change merges. In particular, exercise wrapped calls, bare calls, ordinary prose containing call:, nested braces/string delimiters, incomplete calls, and a marker split across streaming chunks. The last case matters because the new no-marker branch immediately emits all current text; a partial ca/call: prefix can otherwise leak as content before the next chunk turns it into a tool call. The PR test plan currently leaves wrapped regression unchecked and the diff changes parsing behavior without a checked-in test.

The bare `call:` streaming path emitted any no-marker text immediately, so a split `<|tool` / `call` prefix leaked as content before the next chunk completed the marker (CI: start tag split across chunks). Buffer trailing partial `<|tool_call>` and `call:` prefixes on the plain path; do not treat incomplete bare calls (no `{`) as content. Move ScanBareCalls into the anon namespace and share StripTrailingToolMarkerPrefixes with ScanBlocks. Adds maint-bot regressions: wrapped, bare, prose `call:`/callback:, nested braces+delims, incomplete, stream splits for start-tag and bare `call:`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Hermes:grok-4.5 [Hermes]

bakon11 · 2026-08-11T04:23:47Z

@localai-org-maint-bot Addressed CHANGES_REQUESTED:

Bugfix — streaming no-marker branch no longer emits trailing partial <|tool_call> / bare call: prefixes (the CI failure gemma4 stream: start tag split across chunks buffers cleanly). Incomplete bare calls without { are buffered, not leaked as content.
Regressions in tests/.../test_gemma4.cpp:
- non-stream: wrapped, bare complete, prose prefix, callback: prose (not a tool), nested braces + string delims, incomplete bare
- stream: bare complete, call: split across chunks, prose-then-bare, incomplete call buffer, start-tag split (extra coverage)

Local: test_tool_parser_gemma4 69/69 (was 58 + new cases; previously 1 fail on tip).

…ler#328) Free-form / detokenized Gemma-4 output often carries the tool body without the `<|tool_call>` specials, so the wrapped-only scanner returned it as plain content and the request never produced tool_calls. The gemma4 parser gains a bare `call:NAME{ARGS}` scanner (brace depth, `<|"|>` string-aware) used when no wrapped marker is present, in both the complete and streaming paths, and OpenAI chat routes `gemma4` to the text seam so both shapes extract. The branch's two commits land as one because check-doc-checkpoint is a per-commit obligation and the second moved the parser without its USAGE.md line; the marker-buffering work and its regressions are Don Mirror's, unchanged apart from the repair below. docs/USAGE.md keeps main's parser counts (41 names / 37 families) and `muse_glimmer` in the reasoning list, both of which landed after this branch forked, and takes the PR's gemma4 sentence. REPAIR APPLIED ON LANDING: the partial-marker buffer was widened from `<|tool_call>` prefixes to `call:` prefixes unconditionally, so a streamed reply ending in "c", "ca", "cal" or "call" had those characters held back — and this seam has NO end-of-stream flush, so held-back text is never emitted. Any reply ending in "topic", "music" or "specific" would silently lose its last letter. The buffer now applies the same boundary rule ScanBareCalls already uses (not preceded by an identifier character or '/'), so a partial inside a word streams through while a genuine split marker after whitespace is still buffered. test_tool_parser_gemma4 72/72 (69 before; 3 added) The three added cases are mutation-checked: removing the boundary guard fails two of them on the swallowed trailing letter. KNOWN LIMIT, not introduced here and not closed here: bare-call detection is heuristic. Text of the form "... call:name{...}" in ordinary prose becomes a tool call. The wrapped path is unambiguous; the bare path is a lab/Hermes affordance, reachable only when tools are enabled with --tool-call-parser gemma4. Refs mudler#328. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:deepseek-v4-flash [edit bash]

localai-org-maint-bot requested changes Aug 11, 2026

View reviewed changes

Navigation Menu

fix(server): gemma4 bare call: tool extract for OpenAI chat - #328