Skip to content

feat(headless): add the Reasonix benchmark arm - #2079

Merged
Astro-Han merged 6 commits into
mainfrom
eval/headless-reasonix-arm
Aug 4, 2026
Merged

feat(headless): add the Reasonix benchmark arm#2079
Astro-Han merged 6 commits into
mainfrom
eval/headless-reasonix-arm

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds Reasonix (esengine/DeepSeek-Reasonix) as a competitor arm so Maka can be benchmarked head-to-head on deepseek-v4-flash. Refs #2076.

Reasonix is DeepSeek-native, which is what makes it the closest available comparison, but the harness had no way to run it: no pinned toolchain, no Harbor adapter, no registry entry. This lands the infrastructure on the existing two-arm path rather than a new one. Scheduling, resume identity, manifests, pricing, deadline policy, verifier, and report rendering are reused unchanged; the registry needs only an id and an import path, because DeepSeek already resolves to bearer auth and openai-chat-sse usage.

The first commit is a no-op refactor: each pinned arm was wired by hand in five places, and the mount ternary was already four levels deep. It collapses those into one table and moves the toolchain preparer, path-override env key, and runner option onto the competitor profile, so the profile is the single source of truth instead of being re-dispatched downstream. Splitting it out keeps the behaviour-preserving change reviewable and revertable on its own.

Three decisions worth flagging:

  • Pinned Node as verifier only. Reasonix ships a static Go binary in @reasonix/cli-linux-x64, whose tarball layout is identical to opencode-linux-x64. It reuses the shared node-cli-toolchain implementation, and the pinned Node next to it is only the offline manifest verifier install() runs — exactly the role it already plays for the prebuilt OpenCode binary. Making node optional in the shared implementation would not perturb existing fingerprints (each is sha256 of its own spec constant), but it would widen the verification surface to save one cached download.
  • stream-json, not --events-jsonl. internal/cli/run_output.go:265-270 passes tool names through machineOpaqueValue, so the redacted surface aliases real tool names away. All four existing competitor arms capture their CLI's native stream, and the arms are only comparable on tool behaviour if this one does too. The secrets that need protecting — the upstream key and the proxy token — never appear in the event stream either way.
  • Pier is excluded from PierAgent. Terminal-Bench runs under plain Harbor. Letting the type admit an arm that has no mount would be a half-supported state.

Secret boundary

Reasonix has no flag or env override for a provider base_url (verified by grep over internal/ and cmd/), so the proxy route must come from config. It is written to an isolated $REASONIX_HOME/config.toml rather than the task workspace, and the key is referenced indirectly through api_key_env.

api_key_env does not read the process environment: ProviderEntry.APIKey() resolves it through storedCredentialValue(), which reads $REASONIX_HOME/.env and nothing else (internal/config/credentials.go:537, internal/config/paths.go:406). The token therefore has to be materialised in that file. It is written under umask 077 by shell expansion of the already-exported variable, so the value never appears in the command line, in config.toml, in the task workspace, or in any log. The credential proxy remains the authoritative source of token, cache, and cost evidence; Reasonix's self-reported total_cost is not used for settlement (its total_cost_usd alias is not converted when currency is CNY).

Verification

  • npm --workspace @maka/headless run test (clean build + full suite): 1344 tests, 0 failures. The stream-verdict fix was reverse-verified: reverting it fails the suite.
  • npm run format:check, npm run lint: clean.
  • The refactor commit was verified on its own before the feature was applied: 1340 tests, 0 failures — the same suite minus the four new Reasonix tests.

New coverage:

  • reasonix-toolchain.test.ts — pins the tarball integrity and binary SHA-256, and rejects a self-signed cache.
  • harbor-adapter.test.ts — asserts the proxy token is absent from both the command line and the generated config, that reasonix.toml is never written to the workspace, that tool_result does not inflate tool-call counts, and that a self-reported is_error on a zero exit code still lands as runtime_error.
  • harbor-task-runner.test.ts — mount, fingerprint env, no credential serialization, and version-drift refusal.
  • harness-ab-cli.test.ts — composition, run identity, toolchain path, and env override.

Live smoke

One paired cell on configure-git-webserver (Terminal-Bench 2.1 @ d49e28f1), both arms on deepseek-v4-flash at max, against the real DeepSeek key through the credential proxy:

arm outcome scored tokens (cached) cost
maka passed yes 136 886 (122 624) $0.0036
reasonix passed yes 672 080 (653 952) $0.0064

The stream parser recovered the real session id and 35 tool calls with their real names — {bash: 23, todo_write: 5, complete_step: 5, write_file: 1, wait: 1} — which is the evidence that stream-json was the right surface: --events-jsonl would have aliased every one of those names away.

Two live findings no static review surfaced, both fixed in this branch:

  • The proxy token never reached the CLI. api_key_env is not a process-environment lookup (see above); the run died at provider "maka-proxy/deepseek-v4-flash": missing env MAKA_REASONIX_PROXY_TOKEN. Fixed by writing $REASONIX_HOME/.env.
  • Reasonix had silently disabled its own bash tool. Its default [sandbox] bash = "enforce" requires an OS sandbox and refuses bash outright when none exists; Terminal-Bench images ship no bubblewrap. The arm spent an entire trial writing a script it could never run, and lost the cell to verification_failed — a defect that would have been invisible in an aggregate score and would have understated Reasonix across the board. Pinning bash = "off" restores it to the same tool surface as every other arm, which all run unconfined inside the container that is itself the isolation boundary.

Remaining work

  • Predeclared canary passes for both arms
  • One full synchronized Terminal-Bench 2.1 run
  • docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-reasonix.{md,csv} report checked in

Each pinned competitor arm was wired by hand in five places: a required-path
check, a version check, a nested mount ternary, a fingerprint env assignment,
and two id dispatches in the harness runner. Adding an arm meant touching all
of them, and the mount ternary was already four levels deep.

Collapse the per-adapter branches in harbor-task-runner.ts into one
COMPETITOR_TOOLCHAINS table, and move the toolchain preparer, path-override env
key, and runner option name onto the competitor profile in run-harness-ab.mjs
so the profile stays the single source of truth instead of being re-dispatched
downstream.

No behaviour change: the full headless suite passes unchanged.
Reasonix is DeepSeek-native, which makes it the closest head-to-head
comparison available for Maka on deepseek-v4-flash, but the harness had no way
to run it: no pinned toolchain, no Harbor adapter, and no registry entry.

Add it on the existing two-arm path rather than a new one:

- reasonix-toolchain.ts pins @reasonix/cli-linux-x64 through the shared
  node-cli-toolchain implementation. Reasonix is a static Go binary, so the
  pinned Node next to it is only the offline manifest verifier, the same role
  it plays for the prebuilt OpenCode binary.
- reasonix_agent.py drives `reasonix run --auto --effort max --output-format
  stream-json` on the BaseInstalledAgent seam kimi_code_agent.py established,
  and normalizes the stream into the existing cell contract: tool_dispatch
  events for tool calls, the terminal result object for session identity,
  turns, and self-reported failure.
- The registry needs only an id and an import path: DeepSeek already resolves
  to bearer auth and openai-chat-sse usage, so no agent-specific branch.

Secret boundary: Reasonix has no flag or env override for a provider base_url,
so the proxy route is written to an isolated $REASONIX_HOME/config.toml rather
than the task workspace, and the key is referenced by api_key_env. The
cell-scoped proxy token exists only in the process environment — never a
config file, never a command line, never the task workspace. The credential
proxy stays the authoritative source of token, cache, and cost evidence.

stream-json rather than --events-jsonl: the redacted surface aliases real tool
names away, and the four existing competitor arms all capture their CLI's
native stream, so both arms are only comparable on tool behaviour this way.

Pier is deliberately excluded from PierAgent: Terminal-Bench runs under plain
Harbor, and a type that admits an arm with no mount is a half-supported state.

Scheduling, resume identity, manifests, pricing, deadline policy, verifier, and
report rendering are reused unchanged.

Refs #2076
The first live cell failed with `provider "maka-proxy/deepseek-v4-flash":
missing env MAKA_REASONIX_PROXY_TOKEN`, because `api_key_env` does not mean
what it looks like. `ProviderEntry.APIKey` resolves through
`storedCredentialValue`, which reads only `$REASONIX_HOME/.env` — the process
environment is never consulted (internal/config/credentials.go:537). Setting
the variable in the container environment could therefore never work.

Write the token into that credential file instead, expanded from the
environment inside the command rather than interpolated into it, so the value
still never enters the command line, the process table, or Harbor's command
logs. `umask 077` precedes every write so the file is never even momentarily
group- or world-readable, and it lives in the isolated Reasonix home rather
than the task workspace.

This narrows the earlier claim that the token never touches a file: it now
touches one, inside the container. The boundary that actually matters is
unchanged — the upstream DeepSeek key stays host-side in the proxy and never
enters a container in any form, and the token that does is cell-scoped and
proxy-only, the same exposure every other competitor arm already has.

Reasonix's TOML has no `api_key` field at all, so there is no way to name the
value in config; the credential store is the mechanism the CLI provides.
Review found that the self-reported-failure branch was dead on arrival.
Reasonix binds its verdict to its exit code — every `error_during_execution`
also exits 1 (internal/cli/run_completion.go) — so `run()` has always raised by
the time the cell output is written, and the old `error_class is None` guard
meant the stream's verdict was never read. The live smoke run confirmed it: the
stream said `error_during_execution`, the cell said `infra_failed`.

That is not a cosmetic mislabel. `infra_failed` is an unscored class, so every
genuine Reasonix agent failure would have dropped out of the scored
denominator while Maka's equivalent failures stayed in it — a systematic bias
toward Reasonix in the one comparison this harness exists to make.

Classify from the stream's own message whenever the run reported its own
failure. Harbor's exception text is the worse signal: it embeds the entire
command line, so a task instruction mentioning a socket or a connection can
match the infrastructure markers by chance. Real infrastructure faults are
still recognized, because the marker scan runs against the stream message too;
only the fallthrough differs, and for a self-reported failure it is
`runtime_error`. Unknown future subtypes fail the same safe way rather than
silently shrinking the denominator.

Also from review:

- Delete `_cell_timeout_sec` and its constants. Copied from the Kimi adapter,
  where it feeds a request-timeout knob Reasonix does not have; here it had no
  caller at all.
- Delete `network_allowlist`. It is a Pier-only hook, and `PierAgent` excludes
  Reasonix, so it was unreachable.
- Type `COMPETITOR_TOOLCHAINS` as `Record<Exclude<HarnessAgentId, 'maka'>, …>`
  so a future arm cannot be added without a toolchain and silently run with no
  mount.

The new tests cover the path that actually happens: run() raised, the stream
reported the failure, and the class must come from the stream — including the
noisy-instruction case, the unknown subtype, and the genuine auth/rate-limit/
unavailable cases that must stay infrastructure. Reverting the fix makes them
fail.
Reasonix defaults to [sandbox] bash = "enforce", which requires an OS
sandbox (bubblewrap) and otherwise refuses to run bash at all. Terminal-
Bench images ship none, so the arm spent an entire smoke trial writing a
script it could never execute. The Harbor container is already the
isolation boundary and every other arm runs unconfined inside it, so pin
bash = "off" to put Reasonix on the same tool surface.
A run that wrote a terminal result object reached the model and burned
tokens, so it is an agent outcome whatever the message says. Re-scanning
that message for infrastructure markers reintroduced the very coincidence
the stream-verdict override exists to defeat, this time on prose the model
itself wrote: a task about sockets, or a model narrating that "the
connection kept dropping", classified as network and dropped the cell out
of the scored denominator — biasing the comparison toward Reasonix.

Trust the verdict unconditionally on that path and keep the marker scan
for the one case with nothing to trust: a run that died before writing a
result. That collapses the classifier back to a single function.

Also pin the toolchain fingerprint refusal, which no test reached, and
name the log-download fallback after the condition it actually covers
rather than after Pier, which this arm does not run under.
@Astro-Han
Astro-Han marked this pull request as ready for review August 4, 2026 06:13
@Astro-Han
Astro-Han merged commit 8e7f467 into main Aug 4, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant