fix(headless): bound teardown and mark the output replay - #2151

Astro-Han · 2026-08-04T13:11:10Z

Summary

Two holes in the same teardown, found while tracing a graded cell that produced no reward. Follow-up to #2146, which gave the output replay a teardown handle but left both of these open.

Teardown had no time bound. cleanup_process_scope swallows every failure because teardown is best effort. Time is the one failure swallowing cannot cover: the exec was issued with no timeout_sec, and Harbor's _run_docker_compose_command with no timeout awaits communicate() forever. A teardown that hangs takes the whole trial with it and no reward is ever produced. A graded cell was observed stalled for 1524 s with both teardown execs still alive — nothing in the system would ever have ended it. The bound is what makes best effort true.

The replay's handle was recorded too late. The process group is written after the replay is forked, so a teardown landing between the fork and the write finds nothing to kill and strands the replay exactly as before. No recording can close that window, because the name does not exist until the fork has happened. The scope marker is a name teardown already knows beforehand, so the replay now carries the same marker its command does and the window disappears. The recorded process group stays, because it is all that works on a host with no /proc, where the marker sweep cannot run at all — including the machine the regression test runs on.

Raised independently by three reviewers on #2146 and tracked there.

Refs #1970

Verification

python3 packages/headless/harbor/tests/test_process_scope.py — 7 passed, 0 failed.

With process_scope.py restored from main, the two new tests fail and the rest pass:

FAIL test_every_teardown_exec_is_time_bounded: AssertionError([None, None])
FAIL test_the_replay_carries_the_scope_marker: AssertionError(...)
5 passed, 2 failed

packages/headless/harbor/tests/test_harness_compat.py reports 5 pre-existing failures on this machine (ModuleNotFoundError: No module named 'harbor'); identical on a clean main.

Not run: Biome lint/format and workspace typecheck, which do not cover Python. No TypeScript changed.

Review focus

CLEANUP_TIMEOUT_SEC is 60. Teardown signals process groups and walks /proc once per signal, so seconds is the honest scale and a pass that has not finished inside a minute has already failed at its job. If a real workload needs longer, the number is the thing to argue about — not whether a bound belongs here.

The mechanism that made the production teardown hang is not established. The forensic run that would have shown it was competing with a graded run on the same host and was stopped. What is established is that it did hang, and that nothing bounded it. This PR makes that class of hang survivable rather than explaining this instance of it.

An earlier reading attributed the production Command failed (exit -9) to Harbor's own compose timeout. That was wrong: Harbor's timeout path raises Command timed out after N seconds, and exit -9 comes from BaseInstalledAgent._exec seeing a return code from a CLI that was SIGKILLed externally — by an operator watchdog on the benchmark host, whose log shows kill -9 compose child in the same second. The hang is real; that particular symptom was not evidence of a Harbor timeout.

Two holes in the same teardown, found while tracing a graded cell that produced no reward. Teardown swallows every failure because it is best effort. Time is the one failure swallowing cannot cover: the cleanup exec was issued with no bound, and Harbor with no timeout waits on `communicate()` forever, so a teardown that hangs takes the whole trial with it. A cell was observed stalled for 1524s with both teardown execs still alive; nothing in the system would ever have ended it. A bound is what makes best effort true. The replay's process group is recorded after the replay is forked, so a teardown landing between the fork and the write finds nothing to kill and strands the replay exactly as before. No recording can close that window, because the name does not exist until the fork has happened. The scope marker is a name teardown already knows beforehand, so give the replay the same marker the command carries and the window disappears. The recorded process group stays: it is all that works on a host with no `/proc`, where the marker sweep cannot run at all. Refs #1970

Two teardowns run the scope cleanup shell — the adapters' and the bridged executor's — and both reach it from a `finally` with the agent's own exception in flight. Only one of them got the bound this branch added, so the executor could still stall on `communicate()` forever, which is the 1524s hang it exists to prevent. Give the exec one owner that carries the budget; the callers keep the error handling, which is what differs. Kill the wrapper before sweeping for what the wrapper produces. Reaping the command is what makes the wrapper fork a replay, so a command-first sweep races the wrapper it just woke: the replay can appear after the `/proc` sweep passes and the wrapper die before recording a pgid, stranding a replay that no handle describes and that still holds the caller's stdout. A command that ignores SIGTERM survives to die on exactly the pass where that bites. Neither handle closes this alone, so the ordering does.

Marking each `cat` left their parent anonymous, and the parent is the one that decides there is a second `cat`. A sweep kills the first, the parent survives it and starts the stderr replay, and the sweep has already walked past that pid. On the last pass nothing follows, so that second `cat` holds the caller's stdout open for good — the hang this branch exists to end. One marked `sh` owns both replays instead; its children inherit the marker. Pinning the helper was not the same as pinning its use: restoring the one unbounded line at the bridged executor left every test green. `ast` reads maka_agent without importing it, which the stdlib-only CI contract requires, and fails if any teardown reaches the transport without the shared budget. Also stop failing where `ps` exists but a sandbox refuses it.

Three mutations survived the suite. `CLEANUP_TIMEOUT_SEC = None` satisfied every assertion written against the constant and handed Harbor back the unbounded `communicate()`; the budget is now checked for being one before anything is compared to it. The per-command cleanup — the path that cancels a single in-flight exec — had no behavioral test at all, so aiming its wrapper kills at a filename that does not exist changed nothing anyone could see. It has one now, built from `bash`, `sleep` and signal 0 so it runs where there is no `/proc`. Asserting the wrapper is gone is not enough: the same pass deletes the files its replay reads, so a wrapper cleanup never touched also exits, carrying the status of the command it waited on. The signal is what separates them.

Astro-Han added 4 commits August 4, 2026 21:10

Astro-Han marked this pull request as ready for review August 4, 2026 14:26

Astro-Han merged commit 9347038 into main Aug 4, 2026
10 checks passed

This was referenced Aug 4, 2026

eval(headless): benchmark Maka, Codex, and Claude Code synchronously on DeepSeek V4 Flash #1970

Closed

feat(headless): let the Maka arm run where its competitors run #2177

Merged

Navigation Menu

Uh oh!

fix(headless): bound teardown and mark the output replay - #2151