fix(headless): classify a benchmark trial by who ended it, not by how the exception reads - #2051
Merged
Merged
Conversation
… the exception's wording A trial whose agent phase ended abnormally was classified by the exception message: only two pinned regexes counted as a spent budget, and the artifacts that prove whether the trial was actually graded were read only after one of them matched. Everything else hit the `exitCode !== 0` gate and became an infra failure — discarded from the Pass@1 denominator even though Harbor's single_step.py and pier's trial.py both run the verifier after any agent-phase exception, so the reward was already on disk. Read the artifacts for every exception shape and let the verdict decide. A graded trial with a non-budget exception now scores on its real reward; the budget-shaped path is unchanged, and a verifier that reached no verdict (candidate_timeout / ungraded / missing artifacts) still falls through to budget or infra as before. Nothing fabricates a deadline settlement for a non-budget exception — the structured verifier grade already makes the trial scoreable, so the cell keeps its own status and errorClass. The infra message now names the trial exception. The WAL records only the message, which is why 225 infra events carried no trace of the timeouts inside them. Refs #1970
…tion reads The previous commit read the artifacts for every exception shape, but still decided budget-vs-infra from the exception's prose and still let the exit code overrule a graded trial. Both are re-derivations of a fact that arrives typed: Harbor records `exception_type` as Python's `type(e).__name__`, and readTrialException was flattening it into `"Type: message"` before anyone could use it. Keep it structured, and classify on it. Who ended the agent phase and whether the verifier reached a verdict are now the only two inputs: - agent_budget (Harbor's own AgentTimeoutError, or the host cell's deadline — the one case still matched by message, because this repo raises it and harbor-adapter.test.ts pins both ends): settles by deadline as before. - agent_exit (NonZeroAgentExitCodeError): the agent's own process ended it, so a verdict on the workspace it left is a real result, whatever the exit code. - external (Docker, the harness, anything unrecognized): the agent never got the run it was given, so no artifact makes it a fair sample. Stays infra. That last row closes the trade-off the previous commit opened: a container that dies after the verifier ran no longer reads as a scored zero. It also removes the wording-drift class outright rather than widening what survives it. The terminal-provider-request gate had the same defect in miniature — a boolean "this trial settled" exempted every non-completing tail request. Only a client-side teardown (`aborted`) is explained by the agent phase ending; `interrupted` and `failed` are the upstream's doing and stay infra however the trial settled. That keeps a provider outage out of the denominator instead of recording it as the agent's zero, and it now applies to both settlement modes. Claude Code's adapter never emitted deadlineSettlement at all — not a missing assignment but a missing field — so the same timeout was labelled budget_exhausted for Codex and a plain runtime failure for Claude Code. Both arms now report it, pinned by a contract test: settlement must not depend on which adapter happened to record the fact. Refs #1970
…e seam The arm-symmetry check asserted against Python source text with regexes while harbor-adapter.test.ts already had the right seam a few hundred lines away: cancel_codex() constructs the real adapter, cancels its agent phase, and reads the cell output it wrote. A source regex cannot see semantics — moving the flag into `finally` (claiming a benchmark deadline on every successful run) or inverting the emission guard both satisfy the patterns while inverting what the arm reports. Mirror cancel_codex() in the Claude Code smoke script instead, and add the negative half neither arm had: a normal run must not claim a deadline settled it, which an unconditional write would otherwise pass. Also scope the external-termination test's claim to what it actually pins. Both harnesses swallow a recorded agent-phase exception and exit 0, so "infra however complete its artifacts are" was only true on the non-zero-exit path. Refs #1970
Astro-Han
marked this pull request as ready for review
August 3, 2026 19:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A benchmark trial whose agent phase ended abnormally was classified by the wording of its exception.
isBudgetExhaustedTrialExceptionmatched two pinned messages; only on a match did the runner read the reward / verifier outcome / cell output. Everything else hit theexitCode !== 0gate and was thrown as infra — dropped from the Pass@1 denominator — even though both harnesses run the verifier after an agent-phase exception, so the reward was already on disk.The root defect is narrower than "the regexes are brittle": the termination reason arrives typed and was being thrown away. Harbor records
exception_typeas Python'stype(e).__name__(harbor/models/trial/result.py:31), andreadTrialExceptionflattened it to"Type: message"before any caller could use it. Every downstream decision — the regexes, the exit-code gate, the settle predicate — was re-deriving that discarded fact by matching prose.So keep it structured. Classification now reads two facts, who ended the agent phase and whether the verifier reached a verdict:
AgentTimeoutError, host-cell deadline,NonZeroAgentExitCodeError)budget_exhaustedagent_budget— Harbor's ownAgentTimeoutErrorclass is unambiguous by type. The host cell's deadline arrives as a genericRuntimeError, so it is the one termination still matched by message; that is sound only because this repo raises it (harbor/maka_agent.py) andharbor-adapter.test.tspins both ends together.agent_exit—NonZeroAgentExitCodeErroris the agent's own process ending, so a verdict on the workspace it left is a real result whatever the exit code. Nothing fabricates adeadlineSettlementfor it;taskCompletedScoringProjectionalready scores onverifierGrade !== undefined, so the cell keeps its honeststatus/errorClass.external— unrecognized types land here deliberately. The recognized budget shapes are matched by type, so nothing regresses when upstream rewords a message.The terminal-provider-request gate had the same defect in miniature. A boolean "this trial settled" exempted every non-completing tail request. Only a client-side teardown (
aborted, set fromsignal.aborted) is explained by the agent phase ending —interruptedandfailedare the upstream's doing and now stay infra however the trial settled. This tightens the pre-existing budget path in the safe direction: a provider outage is excluded from the denominator rather than recorded as the agent's zero.Arm symmetry.
claude_code_agent.pynever emitteddeadlineSettlement— not a missing assignment but a missing field in_write_cell_output— so the same termination was labelledbudget_exhaustedfor Codex and a plain runtime failure for Claude Code. Both arms now report it, pinned by extending the existingcancel_codex()behavioral seam rather than by matching adapter source text.Infra errors now name the trial exception in their message. The WAL persists only
error: errorMessage(...)and dropsdetail, which is why 225 infra events in the current run carried no trace of the timeouts inside them.Refs #1970. Independent of #2047.
Verification
npm run test -w @maka/headless— 1337 pass / 1 skip, three consecutive runs (baseline 1326 pass / 1 skip).npm run lint,npm run format— clean.classifyTrialTermination; both conjuncts of the settle predicate; the terminal gate exempting everything / exempting nothing; pier settling external types; and the three adapter mutations a source regex could not see (flag moved intofinally, emission guard inverted, emission deleted).Review focus
Six independent reviews (deepseek-v4-flash via
pi; two rounds × design / correctness / test-quality lenses) returned no P0 or P1. The round-one convergent finding — that the terminal-provider-request gate silently undid the fix — is addressed above. Round two's strongest finding was that the arm-symmetry test was a source-text proxy when an executable seam already existed; that is now fixed, and the two semantic mutations that defeated the regex are pinned.Known and deliberately not addressed here:
externalis narrower than it looks. Both harnesses catch only(AgentTimeoutError, NonZeroAgentExitCodeError)in_run_agentand reach_run_verifier()only from there (harbor/trial/single_step.py:75-86,trial/trial.py:286-305); any other exception routes to_recover_outputs(), which does not run the verifier. So a conclusive verdict on disk is already evidence of an agent-owned termination, and both harnesses exit 0 on a recorded agent-phase exception. Theexternalrow therefore fires mainly on the non-zero-exit path. Collapsing it — letting evidence decide scoring, telemetry decide fairness, and type decide only the label — is the smaller structure, but it is a fourth restructure on the eve of a full rerun and is not worth the risk tonight.NonZeroAgentExitCodeErrorhas a subclass hierarchy (ApiRateLimitError,ContextWindowExceededError,AgentSafetyRefusalError, …) thattype(e).__name__records as leaf names, soagent_exitdoes not reach them. Not a regression — they behave exactly as onmain— but the improvement does not reach the Pier arms. The collapse above would fix this for free.aborteddiscriminator is event-order dependent. If a connection dies in the same instant as the agent kill, the proxy may recordfailed/interruptedbeforesignal.abortedis set. The right owner is the proxy (classify teardown by client-connection state, not by which error arrives first), not the runner gate.harbor/maka_agent.pystill raises a genericRuntimeErrorfor the host-cell deadline, so one message regex survives. A distinct exception class would remove it, but that is a cross-language contract change and a mistake there silently reclassifies timeouts as external.