feat(headless): add the Reasonix benchmark arm - #2079
Merged
Conversation
Each pinned competitor arm was wired by hand in five places: a required-path check, a version check, a nested mount ternary, a fingerprint env assignment, and two id dispatches in the harness runner. Adding an arm meant touching all of them, and the mount ternary was already four levels deep. Collapse the per-adapter branches in harbor-task-runner.ts into one COMPETITOR_TOOLCHAINS table, and move the toolchain preparer, path-override env key, and runner option name onto the competitor profile in run-harness-ab.mjs so the profile stays the single source of truth instead of being re-dispatched downstream. No behaviour change: the full headless suite passes unchanged.
Reasonix is DeepSeek-native, which makes it the closest head-to-head comparison available for Maka on deepseek-v4-flash, but the harness had no way to run it: no pinned toolchain, no Harbor adapter, and no registry entry. Add it on the existing two-arm path rather than a new one: - reasonix-toolchain.ts pins @reasonix/cli-linux-x64 through the shared node-cli-toolchain implementation. Reasonix is a static Go binary, so the pinned Node next to it is only the offline manifest verifier, the same role it plays for the prebuilt OpenCode binary. - reasonix_agent.py drives `reasonix run --auto --effort max --output-format stream-json` on the BaseInstalledAgent seam kimi_code_agent.py established, and normalizes the stream into the existing cell contract: tool_dispatch events for tool calls, the terminal result object for session identity, turns, and self-reported failure. - The registry needs only an id and an import path: DeepSeek already resolves to bearer auth and openai-chat-sse usage, so no agent-specific branch. Secret boundary: Reasonix has no flag or env override for a provider base_url, so the proxy route is written to an isolated $REASONIX_HOME/config.toml rather than the task workspace, and the key is referenced by api_key_env. The cell-scoped proxy token exists only in the process environment — never a config file, never a command line, never the task workspace. The credential proxy stays the authoritative source of token, cache, and cost evidence. stream-json rather than --events-jsonl: the redacted surface aliases real tool names away, and the four existing competitor arms all capture their CLI's native stream, so both arms are only comparable on tool behaviour this way. Pier is deliberately excluded from PierAgent: Terminal-Bench runs under plain Harbor, and a type that admits an arm with no mount is a half-supported state. Scheduling, resume identity, manifests, pricing, deadline policy, verifier, and report rendering are reused unchanged. Refs #2076
The first live cell failed with `provider "maka-proxy/deepseek-v4-flash": missing env MAKA_REASONIX_PROXY_TOKEN`, because `api_key_env` does not mean what it looks like. `ProviderEntry.APIKey` resolves through `storedCredentialValue`, which reads only `$REASONIX_HOME/.env` — the process environment is never consulted (internal/config/credentials.go:537). Setting the variable in the container environment could therefore never work. Write the token into that credential file instead, expanded from the environment inside the command rather than interpolated into it, so the value still never enters the command line, the process table, or Harbor's command logs. `umask 077` precedes every write so the file is never even momentarily group- or world-readable, and it lives in the isolated Reasonix home rather than the task workspace. This narrows the earlier claim that the token never touches a file: it now touches one, inside the container. The boundary that actually matters is unchanged — the upstream DeepSeek key stays host-side in the proxy and never enters a container in any form, and the token that does is cell-scoped and proxy-only, the same exposure every other competitor arm already has. Reasonix's TOML has no `api_key` field at all, so there is no way to name the value in config; the credential store is the mechanism the CLI provides.
Review found that the self-reported-failure branch was dead on arrival. Reasonix binds its verdict to its exit code — every `error_during_execution` also exits 1 (internal/cli/run_completion.go) — so `run()` has always raised by the time the cell output is written, and the old `error_class is None` guard meant the stream's verdict was never read. The live smoke run confirmed it: the stream said `error_during_execution`, the cell said `infra_failed`. That is not a cosmetic mislabel. `infra_failed` is an unscored class, so every genuine Reasonix agent failure would have dropped out of the scored denominator while Maka's equivalent failures stayed in it — a systematic bias toward Reasonix in the one comparison this harness exists to make. Classify from the stream's own message whenever the run reported its own failure. Harbor's exception text is the worse signal: it embeds the entire command line, so a task instruction mentioning a socket or a connection can match the infrastructure markers by chance. Real infrastructure faults are still recognized, because the marker scan runs against the stream message too; only the fallthrough differs, and for a self-reported failure it is `runtime_error`. Unknown future subtypes fail the same safe way rather than silently shrinking the denominator. Also from review: - Delete `_cell_timeout_sec` and its constants. Copied from the Kimi adapter, where it feeds a request-timeout knob Reasonix does not have; here it had no caller at all. - Delete `network_allowlist`. It is a Pier-only hook, and `PierAgent` excludes Reasonix, so it was unreachable. - Type `COMPETITOR_TOOLCHAINS` as `Record<Exclude<HarnessAgentId, 'maka'>, …>` so a future arm cannot be added without a toolchain and silently run with no mount. The new tests cover the path that actually happens: run() raised, the stream reported the failure, and the class must come from the stream — including the noisy-instruction case, the unknown subtype, and the genuine auth/rate-limit/ unavailable cases that must stay infrastructure. Reverting the fix makes them fail.
Reasonix defaults to [sandbox] bash = "enforce", which requires an OS sandbox (bubblewrap) and otherwise refuses to run bash at all. Terminal- Bench images ship none, so the arm spent an entire smoke trial writing a script it could never execute. The Harbor container is already the isolation boundary and every other arm runs unconfined inside it, so pin bash = "off" to put Reasonix on the same tool surface.
A run that wrote a terminal result object reached the model and burned tokens, so it is an agent outcome whatever the message says. Re-scanning that message for infrastructure markers reintroduced the very coincidence the stream-verdict override exists to defeat, this time on prose the model itself wrote: a task about sockets, or a model narrating that "the connection kept dropping", classified as network and dropped the cell out of the scored denominator — biasing the comparison toward Reasonix. Trust the verdict unconditionally on that path and keep the marker scan for the one case with nothing to trust: a run that died before writing a result. That collapses the classifier back to a single function. Also pin the toolchain fingerprint refusal, which no test reached, and name the log-download fallback after the condition it actually covers rather than after Pier, which this arm does not run under.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Reasonix (esengine/DeepSeek-Reasonix) as a competitor arm so Maka can be benchmarked head-to-head on
deepseek-v4-flash. Refs #2076.Reasonix is DeepSeek-native, which is what makes it the closest available comparison, but the harness had no way to run it: no pinned toolchain, no Harbor adapter, no registry entry. This lands the infrastructure on the existing two-arm path rather than a new one. Scheduling, resume identity, manifests, pricing, deadline policy, verifier, and report rendering are reused unchanged; the registry needs only an id and an import path, because DeepSeek already resolves to bearer auth and
openai-chat-sseusage.The first commit is a no-op refactor: each pinned arm was wired by hand in five places, and the mount ternary was already four levels deep. It collapses those into one table and moves the toolchain preparer, path-override env key, and runner option onto the competitor profile, so the profile is the single source of truth instead of being re-dispatched downstream. Splitting it out keeps the behaviour-preserving change reviewable and revertable on its own.
Three decisions worth flagging:
@reasonix/cli-linux-x64, whose tarball layout is identical toopencode-linux-x64. It reuses the sharednode-cli-toolchainimplementation, and the pinned Node next to it is only the offline manifest verifierinstall()runs — exactly the role it already plays for the prebuilt OpenCode binary. Makingnodeoptional in the shared implementation would not perturb existing fingerprints (each issha256of its own spec constant), but it would widen the verification surface to save one cached download.stream-json, not--events-jsonl.internal/cli/run_output.go:265-270passes tool names throughmachineOpaqueValue, so the redacted surface aliases real tool names away. All four existing competitor arms capture their CLI's native stream, and the arms are only comparable on tool behaviour if this one does too. The secrets that need protecting — the upstream key and the proxy token — never appear in the event stream either way.PierAgent. Terminal-Bench runs under plain Harbor. Letting the type admit an arm that has no mount would be a half-supported state.Secret boundary
Reasonix has no flag or env override for a provider
base_url(verified by grep overinternal/andcmd/), so the proxy route must come from config. It is written to an isolated$REASONIX_HOME/config.tomlrather than the task workspace, and the key is referenced indirectly throughapi_key_env.api_key_envdoes not read the process environment:ProviderEntry.APIKey()resolves it throughstoredCredentialValue(), which reads$REASONIX_HOME/.envand nothing else (internal/config/credentials.go:537,internal/config/paths.go:406). The token therefore has to be materialised in that file. It is written underumask 077by shell expansion of the already-exported variable, so the value never appears in the command line, inconfig.toml, in the task workspace, or in any log. The credential proxy remains the authoritative source of token, cache, and cost evidence; Reasonix's self-reportedtotal_costis not used for settlement (itstotal_cost_usdalias is not converted whencurrencyisCNY).Verification
npm --workspace @maka/headless run test(clean build + full suite): 1344 tests, 0 failures. The stream-verdict fix was reverse-verified: reverting it fails the suite.npm run format:check,npm run lint: clean.New coverage:
reasonix-toolchain.test.ts— pins the tarball integrity and binary SHA-256, and rejects a self-signed cache.harbor-adapter.test.ts— asserts the proxy token is absent from both the command line and the generated config, thatreasonix.tomlis never written to the workspace, thattool_resultdoes not inflate tool-call counts, and that a self-reportedis_erroron a zero exit code still lands asruntime_error.harbor-task-runner.test.ts— mount, fingerprint env, no credential serialization, and version-drift refusal.harness-ab-cli.test.ts— composition, run identity, toolchain path, and env override.Live smoke
One paired cell on
configure-git-webserver(Terminal-Bench 2.1 @d49e28f1), both arms ondeepseek-v4-flashatmax, against the real DeepSeek key through the credential proxy:The stream parser recovered the real session id and 35 tool calls with their real names —
{bash: 23, todo_write: 5, complete_step: 5, write_file: 1, wait: 1}— which is the evidence thatstream-jsonwas the right surface:--events-jsonlwould have aliased every one of those names away.Two live findings no static review surfaced, both fixed in this branch:
api_key_envis not a process-environment lookup (see above); the run died atprovider "maka-proxy/deepseek-v4-flash": missing env MAKA_REASONIX_PROXY_TOKEN. Fixed by writing$REASONIX_HOME/.env.[sandbox] bash = "enforce"requires an OS sandbox and refuses bash outright when none exists; Terminal-Bench images ship nobubblewrap. The arm spent an entire trial writing a script it could never run, and lost the cell toverification_failed— a defect that would have been invisible in an aggregate score and would have understated Reasonix across the board. Pinningbash = "off"restores it to the same tool surface as every other arm, which all run unconfined inside the container that is itself the isolation boundary.Remaining work
docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-reasonix.{md,csv}report checked in