fix(headless): stop misfiling graded competitor cells as infrastructure failures - #2090

Astro-Han · 2026-08-04T05:56:45Z

Summary

Two independent defects in the Harbor cell path recorded graded cells as infra_failed, which drops them from the denominator entirely. Both are structurally unreachable on the maka arm, so every cell they cost belonged to a competitor.

Found while tracking the three-arm run for #1970. Of that run's 14 infra_failed cells, 13 were these two bugs and 1 was a real infrastructure failure.

Teardown rewrote the agent's failure. cleanup_process_scope only ever runs from an adapter's finally while the agent's own exception is in flight (claude_code_agent.py, codex_agent.py, kimi_code_agent.py all guard it with if abnormal_exit). Raising there replaces the exception being propagated, so an AgentTimeoutError reached Harbor as NonZeroAgentExitCodeError and the runner read it as infrastructure rather than a graded timeout. Teardown after a failure is best effort by nature — no caller can act on its error — so it now logs and continues. Cancellation still propagates. Cost: 10 Claude Code cells, each already carrying errorClass=budget_exhausted in its own cell output while the WAL said infra_failed. maka_agent.py never calls this teardown.

The aborted tail exemption could not fire on a clean exit. incompleteTerminalProviderRequest exempts an aborted last request when the agent phase settled, but the caller computed agentPhaseSettled only inside the abnormal-termination branch. A trial that raised nothing never reached it — so the exemption missed the most ordinary shape there is: an agent that finishes and exits while a stream is open. Cost: 3 codex cells with verifier reward 1.0 and no trial exception, discarded as infrastructure failures. maka produced no aborted tail request anywhere in the run.

Refs #1970.

Verification

npm run --workspace @maka/headless test — 97/97 in harbor-task-runner.test.ts, including the new case. One pre-existing failure remains in cli.test.js (components[0].sourceRefs[2].path: source ref "apps/desktop/src/main/workspace-instructions.ts" does not exist under repo root); it reproduces on unmodified origin/main and is unrelated to this change.
python3 packages/headless/harbor/tests/test_process_scope.py — 3 passed. This directory is run by hand, as test_harness_compat.py already is; it is not wired into CI.
npm run format / npm run lint — clean.

Both tests were confirmed red against the unfixed code, and the TS one fails with the exact production error:

✖ scores a cell whose agent exited cleanly while a provider stream was open
  Error [HarborInfraError]: terminal provider request did not complete for task task-1

FAIL test_a_failing_teardown_never_replaces_the_agent_failure: RuntimeError('Command failed (exit -9)')

Review focus

The fix stops the misfiling going forward; it does not repair WALs already written. The #1970 run's own numbers still need a re-scoring pass from the on-disk artifacts, which will land with that report rather than here.

cleanup_process_scope only ever runs from an adapter's finally while the agent's own exception is in flight, so the exception it raised replaced the one being propagated. An AgentTimeoutError reached Harbor as NonZeroAgentExitCodeError, and the runner read that as an infrastructure failure rather than a graded timeout. In the #1970 three-arm run this cost 10 Claude Code cells, every one of them carrying errorClass=budget_exhausted in its own cell output — the true cause was already on disk while the WAL recorded infra_failed. The maka adapter never calls this teardown, so the failure mode could only land on competitor arms. Teardown is best effort: log and continue. Cancellation still propagates.

incompleteTerminalProviderRequest exempts an `aborted` tail request when the agent phase settled, but the caller only computed agentPhaseSettled inside the abnormal-termination branch. A trial that raised nothing never reached it, so the exemption could not fire on the most ordinary shape of all: an agent that finishes its work and exits while a stream is still open. Three codex cells in the #1970 run were discarded this way with verifier reward 1.0 and no trial exception — passing cells recorded as infrastructure failures. maka produced no aborted tail request in the whole run, so this too could only land on competitor arms. A trial with no exception settled its own agent phase; say so at both call sites.

Three gaps an external review surfaced, all of which let a revert of the fixes ship green: - The pier runner's half of the aborted-tail change had no test at all; the existing terminal-request case only covered `interrupted` (infra either way) and `completed` (graded either way), neither of which distinguishes the hunk. - The Python teardown contract ran nowhere. The claim that this directory is hand-run does not survive contact: `test_harness_compat.py` stays out of CI because it needs Harbor or Pier installed, while `test_process_scope.py` imports only the stdlib, and the headless lane already installs Harbor. - `agent.logger.warning` sat unguarded inside the except handler, so a caller without a logger — or a logging handler that raises — would escape the same `finally` and rewrite the same agent failure the fix exists to preserve. The test double always carried a logger, so nothing could catch it. Both new tests were confirmed red against their unfixed code.

Astro-Han added 2 commits August 4, 2026 14:15

Astro-Han force-pushed the fix/headless-infra-failed-misclassification branch from fe4d35b to e73de9e Compare August 4, 2026 06:16

Astro-Han marked this pull request as ready for review August 4, 2026 06:34

Astro-Han merged commit 2b3b654 into main Aug 4, 2026
11 checks passed

This was referenced Aug 4, 2026

eval(headless): benchmark Maka, Codex, and Claude Code synchronously on DeepSeek V4 Flash #1970

Closed

feat(headless): let the Maka arm run where its competitors run #2177

Merged

Navigation Menu

Uh oh!

fix(headless): stop misfiling graded competitor cells as infrastructure failures - #2090