Skip to content

fix(headless): classify a benchmark trial by who ended it, not by how the exception reads - #2051

Merged
Astro-Han merged 3 commits into
mainfrom
fix/headless-settle-timeouts-by-evidence
Aug 3, 2026
Merged

fix(headless): classify a benchmark trial by who ended it, not by how the exception reads#2051
Astro-Han merged 3 commits into
mainfrom
fix/headless-settle-timeouts-by-evidence

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

A benchmark trial whose agent phase ended abnormally was classified by the wording of its exception. isBudgetExhaustedTrialException matched two pinned messages; only on a match did the runner read the reward / verifier outcome / cell output. Everything else hit the exitCode !== 0 gate and was thrown as infra — dropped from the Pass@1 denominator — even though both harnesses run the verifier after an agent-phase exception, so the reward was already on disk.

The root defect is narrower than "the regexes are brittle": the termination reason arrives typed and was being thrown away. Harbor records exception_type as Python's type(e).__name__ (harbor/models/trial/result.py:31), and readTrialException flattened it to "Type: message" before any caller could use it. Every downstream decision — the regexes, the exit-code gate, the settle predicate — was re-deriving that discarded fact by matching prose.

So keep it structured. Classification now reads two facts, who ended the agent phase and whether the verifier reached a verdict:

verdict no verdict
agent-owned (AgentTimeoutError, host-cell deadline, NonZeroAgentExitCodeError) scored budget_exhausted
external (Docker, harness, unrecognized) infra (on a non-zero exit) infra
  • agent_budget — Harbor's own AgentTimeoutError class is unambiguous by type. The host cell's deadline arrives as a generic RuntimeError, so it is the one termination still matched by message; that is sound only because this repo raises it (harbor/maka_agent.py) and harbor-adapter.test.ts pins both ends together.
  • agent_exitNonZeroAgentExitCodeError is the agent's own process ending, so a verdict on the workspace it left is a real result whatever the exit code. Nothing fabricates a deadlineSettlement for it; taskCompletedScoringProjection already scores on verifierGrade !== undefined, so the cell keeps its honest status / errorClass.
  • external — unrecognized types land here deliberately. The recognized budget shapes are matched by type, so nothing regresses when upstream rewords a message.

The terminal-provider-request gate had the same defect in miniature. A boolean "this trial settled" exempted every non-completing tail request. Only a client-side teardown (aborted, set from signal.aborted) is explained by the agent phase ending — interrupted and failed are the upstream's doing and now stay infra however the trial settled. This tightens the pre-existing budget path in the safe direction: a provider outage is excluded from the denominator rather than recorded as the agent's zero.

Arm symmetry. claude_code_agent.py never emitted deadlineSettlement — not a missing assignment but a missing field in _write_cell_output — so the same termination was labelled budget_exhausted for Codex and a plain runtime failure for Claude Code. Both arms now report it, pinned by extending the existing cancel_codex() behavioral seam rather than by matching adapter source text.

Infra errors now name the trial exception in their message. The WAL persists only error: errorMessage(...) and drops detail, which is why 225 infra events in the current run carried no trace of the timeouts inside them.

Refs #1970. Independent of #2047.

Verification

  • npm run test -w @maka/headless — 1337 pass / 1 skip, three consecutive runs (baseline 1326 pass / 1 skip).
  • npm run lint, npm run format — clean.
  • Mutation battery, 12 mutations, each caught: every branch of classifyTrialTermination; both conjuncts of the settle predicate; the terminal gate exempting everything / exempting nothing; pier settling external types; and the three adapter mutations a source regex could not see (flag moved into finally, emission guard inverted, emission deleted).

Review focus

Six independent reviews (deepseek-v4-flash via pi; two rounds × design / correctness / test-quality lenses) returned no P0 or P1. The round-one convergent finding — that the terminal-provider-request gate silently undid the fix — is addressed above. Round two's strongest finding was that the arm-symmetry test was a source-text proxy when an executable seam already existed; that is now fixed, and the two semantic mutations that defeated the regex are pinned.

Known and deliberately not addressed here:

  • external is narrower than it looks. Both harnesses catch only (AgentTimeoutError, NonZeroAgentExitCodeError) in _run_agent and reach _run_verifier() only from there (harbor/trial/single_step.py:75-86, trial/trial.py:286-305); any other exception routes to _recover_outputs(), which does not run the verifier. So a conclusive verdict on disk is already evidence of an agent-owned termination, and both harnesses exit 0 on a recorded agent-phase exception. The external row therefore fires mainly on the non-zero-exit path. Collapsing it — letting evidence decide scoring, telemetry decide fairness, and type decide only the label — is the smaller structure, but it is a fourth restructure on the eve of a full rerun and is not worth the risk tonight.
  • Pier's NonZeroAgentExitCodeError has a subclass hierarchy (ApiRateLimitError, ContextWindowExceededError, AgentSafetyRefusalError, …) that type(e).__name__ records as leaf names, so agent_exit does not reach them. Not a regression — they behave exactly as on main — but the improvement does not reach the Pier arms. The collapse above would fix this for free.
  • The aborted discriminator is event-order dependent. If a connection dies in the same instant as the agent kill, the proxy may record failed/interrupted before signal.aborted is set. The right owner is the proxy (classify teardown by client-connection state, not by which error arrives first), not the runner gate.
  • harbor/maka_agent.py still raises a generic RuntimeError for the host-cell deadline, so one message regex survives. A distinct exception class would remove it, but that is a cross-language contract change and a mistake there silently reclassifies timeouts as external.

… the exception's wording

A trial whose agent phase ended abnormally was classified by the exception
message: only two pinned regexes counted as a spent budget, and the artifacts
that prove whether the trial was actually graded were read only after one of
them matched. Everything else hit the `exitCode !== 0` gate and became an infra
failure — discarded from the Pass@1 denominator even though Harbor's
single_step.py and pier's trial.py both run the verifier after any agent-phase
exception, so the reward was already on disk.

Read the artifacts for every exception shape and let the verdict decide. A
graded trial with a non-budget exception now scores on its real reward; the
budget-shaped path is unchanged, and a verifier that reached no verdict
(candidate_timeout / ungraded / missing artifacts) still falls through to
budget or infra as before. Nothing fabricates a deadline settlement for a
non-budget exception — the structured verifier grade already makes the trial
scoreable, so the cell keeps its own status and errorClass.

The infra message now names the trial exception. The WAL records only the
message, which is why 225 infra events carried no trace of the timeouts inside
them.

Refs #1970
…tion reads

The previous commit read the artifacts for every exception shape, but still
decided budget-vs-infra from the exception's prose and still let the exit code
overrule a graded trial. Both are re-derivations of a fact that arrives typed:
Harbor records `exception_type` as Python's `type(e).__name__`, and
readTrialException was flattening it into `"Type: message"` before anyone could
use it.

Keep it structured, and classify on it. Who ended the agent phase and whether
the verifier reached a verdict are now the only two inputs:

- agent_budget (Harbor's own AgentTimeoutError, or the host cell's deadline —
  the one case still matched by message, because this repo raises it and
  harbor-adapter.test.ts pins both ends): settles by deadline as before.
- agent_exit (NonZeroAgentExitCodeError): the agent's own process ended it, so
  a verdict on the workspace it left is a real result, whatever the exit code.
- external (Docker, the harness, anything unrecognized): the agent never got
  the run it was given, so no artifact makes it a fair sample. Stays infra.

That last row closes the trade-off the previous commit opened: a container that
dies after the verifier ran no longer reads as a scored zero. It also removes
the wording-drift class outright rather than widening what survives it.

The terminal-provider-request gate had the same defect in miniature — a boolean
"this trial settled" exempted every non-completing tail request. Only a
client-side teardown (`aborted`) is explained by the agent phase ending;
`interrupted` and `failed` are the upstream's doing and stay infra however the
trial settled. That keeps a provider outage out of the denominator instead of
recording it as the agent's zero, and it now applies to both settlement modes.

Claude Code's adapter never emitted deadlineSettlement at all — not a missing
assignment but a missing field — so the same timeout was labelled
budget_exhausted for Codex and a plain runtime failure for Claude Code. Both
arms now report it, pinned by a contract test: settlement must not depend on
which adapter happened to record the fact.

Refs #1970
@Astro-Han Astro-Han changed the title fix(headless): settle an exceptional trial by its verifier grade, not the exception's wording fix(headless): classify a benchmark trial by who ended it, not by how the exception reads Aug 3, 2026
…e seam

The arm-symmetry check asserted against Python source text with regexes while
harbor-adapter.test.ts already had the right seam a few hundred lines away:
cancel_codex() constructs the real adapter, cancels its agent phase, and reads
the cell output it wrote. A source regex cannot see semantics — moving the flag
into `finally` (claiming a benchmark deadline on every successful run) or
inverting the emission guard both satisfy the patterns while inverting what the
arm reports.

Mirror cancel_codex() in the Claude Code smoke script instead, and add the
negative half neither arm had: a normal run must not claim a deadline settled
it, which an unconditional write would otherwise pass.

Also scope the external-termination test's claim to what it actually pins.
Both harnesses swallow a recorded agent-phase exception and exit 0, so "infra
however complete its artifacts are" was only true on the non-zero-exit path.

Refs #1970
@Astro-Han
Astro-Han marked this pull request as ready for review August 3, 2026 19:57
@Astro-Han
Astro-Han merged commit 90f8c45 into main Aug 3, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant