RFC: Close tool-fidelity gaps vs opencode (fault-tolerant Edit, bounded output, ripgrep Glob, LSP feedback)

[Astro-Han]

Context

Artificial Analysis' Coding Agent Index compares harnesses with the model held constant (Claude Opus 4.7, medium). With the same model, opencode scores higher than Claude Code: composite 65 vs 57, DeepSWE 40 vs 27, Terminal-Bench v2 75 vs 71.

We investigated why, focused on the tool layer, against current upstream sst/opencode@dev (57ce1b9), cross-checked by two independent codex reviews + one glm-5.2 review.

Key finding: the benchmark-relevant advantage is not search speed (both use ripgrep) and not a smarter system prompt (opencode's anthropic.txt is ~Claude Code's). It is concentrated in fault-tolerant editing + bounded/structured tool output + post-edit diagnostics + tool-feedback quality. Exact score attribution needs AA run logs we don't have, so work is prioritized by confidence + ROI, not a promised score delta.

⚠️ Scope — the benchmark does NOT run builtin-tools.ts

Confirmed in-repo (this corrects the original framing):

• Harbor / headless pass@1 runs use the isolated tool set buildIsolatedHeadlessTools (packages/headless/src/harbor-cell.ts:305 -> packages/headless/src/tools.ts:14), not buildBuiltinTools.
  • Isolated Edit is also exact-match (packages/headless/src/tools.ts:313-316, via executor.editFile).
  • Isolated Grep is a JS recursive walk + RegExp (packages/headless/src/tools.ts:335-368) — not ripgrep at all, no gitignore.
• The desktop app filters the runtime builtin Edit out entirely (apps/desktop/src/main/main.ts:589).

Implication: a fix that only touches packages/runtime/src/builtin-tools.ts lands in neither the benchmark nor desktop path. P0 must extract a shared implementation used by headless/src/tools.ts, builtin-tools.ts, and the desktop edit path — otherwise pass@1 won't move. The builtin-tools.ts line numbers below describe the runtime baseline; the parallel isolated set must be covered too.

Evidence: opencode upstream vs maka today

Tool	maka now	opencode upstream	Gap
Edit	exact match only, both paths: split(old_string) count -> error if 0/many -> current.replace (builtin-tools.ts:91-101; isolated headless/tools.ts:313-316). No replaceAll, no file lock	9-strategy replacer cascade (trim / block-anchor+Levenshtein / whitespace / indentation / escape / context-aware ...) + per-file Semaphore	keystone — exact edits fail on whitespace/indent/escape drift
Read	whole file utf8, no default cap, no line-number prefix, no binary guard (builtin-tools.ts:60-68)	line-number prefixes + default 2000 lines / 50KB + binary sniff + "did you mean"	no line refs -> weaker Edit anchors; context blow-up risk
Glob	Node fs/promises glob, not ripgrep, not gitignore-aware (builtin-tools.ts:113-121)	rg --files --glob (parallel, gitignore-aware)	slow + walks node_modules/.git
Grep	runtime: exec rg, --max-count=50, slice 200, bare PATH rg (builtin-tools.ts:133-149). benchmark path: JS walk + RegExp, no rg (headless/tools.ts:335-368)	streaming + early process kill at 100, bundled rg, 2000-char line cap	benchmark path doesn't even use rg
Bash	output > 10MB -> rejects and discards all output (builtin-tools.ts:209-211)	truncate + spill full output to file + keep tail + return outputPath	finished work thrown away -> task fails
Feedback	Edit success {ok,path,replacements}, Write {ok,path,bytes} — no line counts/diff. Failed Bash returns only {error:"命令退出码 N"} to the in-turn model (structured {exitCode,stdout,stderr} goes to session history, not the immediate result). Non-Bash errors = raw Error.message; 4000-char cap applies only to synthetic text errors	enriched success (path + line counts), instructive errors ("match exactly incl whitespace", "add more context"), "did you mean" on Read miss, loop-gate on repeated identical failures	weak models can't tell if an edit landed or how to recover (cf. pawwork #1372 -> duplicate edits)
LSP	none	post-edit diagnostics reflow in Edit/Write/apply_patch (on by default) + standalone navigation tool (experimental flag)	missing
apply_patch	none	present, routed only to GPT-class models (Opus uses Edit)	low priority for Opus path

Proposed work (reprioritized after review)

Both reviewers agreed the original P0 ("fuzzy Edit only") was too narrow: a stuck Edit makes the model resubmit the same failing old_string and drains the whole token budget, and Bash discard-on-overflow fails long-output tasks outright. P0 now bundles the minimum that actually moves pass@1.

P0 — shared fault-tolerant Edit + safety rails

• Extract a shared replace() and apply it in headless/tools.ts (benchmark), builtin-tools.ts (runtime), and the desktop edit path.
• Replacer cascade: exact -> line-trimmed -> block-anchor+Levenshtein -> whitespace-normalized -> indentation-flexible -> escape-normalized -> trimmed-boundary -> context-aware -> multi-occurrence (open-source; opencode sources from cline diff-apply + gemini-cli editCorrector).
• Guards against silent wrong-location edits (the main fuzzy risk — exact-miss errors loudly, fuzzy-miss silently corrupts): unique-match requirement; any multi-candidate fuzzy match -> fail; reject short/low-info old_string (<3-5 chars); reject old_string === new_string; reject disproportionately large spans + binary/oversized files; surface matched_via: exact | fuzzy-N; return matched line range + ±3-5 line snippet on success.
• loop-gate: block/stop repeated identical failing tool calls (trigger = same tool+args failing N times; action = block + force a Read).
• Bash overflow: keep tail + spill full output via existing tool-artifacts + return outputPath, instead of reject-and-discard.
• Minimal recoverable feedback: enrich Edit/Write success (Edited <path> (+X -Y)), make Edit errors instructive, return truncated stdout/stderr to the in-turn Bash result.
• Regression tests: indentation drift, whitespace mismatch, escaped chars, minor middle-line drift, multi-candidate rejection, short-string rejection.

P1 — token efficiency / robustness

• Read: default line+byte cap, line-number prefixes, binary guard, "continue with offset" hint, "did you mean" on miss.
• Grep: switch the benchmark path off JS+RegExp to ripgrep; stream early-termination + total cap, per-line truncation, bundled/fallback rg.
• Glob: ripgrep --files --glob (gitignore-aware) instead of Node fs glob.
• File lock / serialize writes to the same path (opencode per-file Semaphore).

P2 — LSP feedback (larger lift, conditional value)

• Post-edit LSP diagnostics reflow in Edit/Write. Value conditional on a running language server.

Non-goals / open questions

• apply_patch: opencode routes it only to GPT-class models; Opus uses Edit. Not needed for the Opus path.
• Exact weighting: opencode's edge is tools + agent-loop/context/provider-budget (compaction, 32k max-output, beta headers, adaptive effort). Splitting the contribution needs AA harness logs — out of scope.
• maka has adjacent infra (context-budget.ts, tool-artifacts.ts) but it acts at the history-compaction layer, not the tool-execution boundary like opencode's truncate.output().

References (opencode upstream sst/opencode@dev 57ce1b9)

• Edit cascade: packages/opencode/src/tool/edit.ts
• Patch matching (seekSequence): packages/opencode/src/patch/index.ts
• ripgrep stream early-termination: packages/core/src/ripgrep.ts
• per-model tool routing: packages/opencode/src/tool/registry.ts
• output spill: packages/opencode/src/tool/truncate.ts
• Feedback-enrichment precedent: pawwork PR feat(storage): make SQLite the default session metadata store #1372 (enrich edit/write success output with path + line counts)

Investigation: Maka session (opencode tool-design study, cross-checked by 2x codex + 1x glm-5.2 review on upstream). Read-only reviews; external AA/opencode numbers not re-verified.