Skip to content

fix(headless): make the smoke harness reach Maka's settlement window - #2114

Merged
Astro-Han merged 2 commits into
mainfrom
fix/headless-smoke-settlement-window
Aug 4, 2026
Merged

fix(headless): make the smoke harness reach Maka's settlement window#2114
Astro-Han merged 2 commits into
mainfrom
fix/headless-smoke-settlement-window

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

The smoke harness published the Maka cell budget plus its settlement grace on max_timeout_sec. Harbor resolves the agent phase as min(override_timeout_sec ?? task_declared, max_timeout_sec ?? inf) * multiplier (harbor/trial/trial.py, _resolve_timeout_sec), so max_timeout_sec is a ceiling — it can only lower the task's declared timeout, never raise it. For maka-basic on the default 900s task that folded to min(900, 3630) * 4 = 3600, the cell budget exactly, and the cell was SIGKILLed at the instant it stopped calling the model and began writing maka-cell-output.json. The symptom packages/headless/src/maka-settlement.ts describes.

#2107 fixed the same bug on the fixed-prompt batch path and deliberately left this one out, because the two paths do not resolve the phase the same way: harbor-task-runner.ts sets no agent_timeout_multiplier, while the smoke profiles do (maka-heavy is 8). Harbor applies that multiplier after the min(), so moving an absolute value onto override_timeout_sec alone would have it multiplied again — 7230 × 8 = 57840.

So this publishes the phase on override_timeout_sec and normalises agent_timeout_multiplier to 1.0 wherever an absolute phase is published. The multiplier's only job is stretching a task-declared timeout into the budget the profile wants, and an absolute value has already absorbed it. 1.0 rather than null: null makes Harbor fall back to the job-level timeout_multiplier, which would rescale the number we just published. Profiles with no cell budget of their own (opencode, oracle) keep their multiplier untouched.

That also drops a premise the old comment took for granted. It claimed the profiles tune the multiplier so that declared * multiplier == cell budget. The multipliers were tuned against a 900s declared timeout alone, but the dataset declares 600/900/1800/3600 (52/92 tasks at 900), so the identity was never true in general — on a 3600s task maka-heavy was resolving a 28800s phase against a 7200s budget. The phase now tracks MAKA_CELL_TIMEOUT_SEC on every task, which is what maka-settlement.ts says the one rule is.

Refs #2107.

Second commit: one model of Harbor's rule, not two

#2107 landed a harborAgentPhaseSec helper in harbor-task-runner.test.ts; this PR had grown its own. Two hand-written copies of one external system's rule is how this bug class survives — if Harbor changes _resolve_timeout_sec, both suites stay green and both are wrong. It now lives once in packages/headless/src/__tests__/helpers/harbor-agent-phase.ts. The shared version models the multiplier, which the fixed-prompt copy deliberately did not, so that suite gets strictly more faithful assertions.

Verification

npm test --workspace=packages/headless (1353 tests, 0 fail), npm run lint, npm run format:check.

Assertions go through the phase Harbor actually resolves, not a field value — a tail on max_timeout_sec satisfies a field-shaped assertion while folding straight back to the task's own timeout, which is how this stayed green while the window was unreachable. Reverting only harbor-smoke-config.ts fails the regression tests and leaves the rest green. Regressing agent_timeout_multiplier from 1.0 to null — invisible to every test before the second commit — now fails two.

Live smoke run

Unit tests cannot show a SIGKILL, so both geometries were run through the real harness (Harbor + Docker + DeepSeek, *sqlite-with-gcov). The shipped maka-basic geometry is budget 3600 / declared 900 / ×4 → phase 3600 = budget. Reproduced at 1/20 the wall clock as budget 180 / declared 900 / ×6⁄7 → phase 180 = budget; the "after" arm is the generated config unmodified.

before (phase = budget = 180s) after (phase = budget + grace = 210s)
config max_timeout_sec: 210, override: null, mult 6⁄7 override_timeout_sec: 210, max: null, mult 1.0
cell wall clock killed at 180s 182.0s, exited budget_exhausted (rc 124)
maka-harbor.status.json frozen at "status": "running" budget_exhausted
maka-harbor.stdout.json 0 bytes — no envelope 55972 bytes, settledByDeadline: true, taxonomy: budget_exhausted, benchmarkFailureShouldThrow: false
Harbor trial AgentTimeoutError, n_errored_trials: 1 no exception, n_errors: 0
reward 1.0 — a verifier-passing run filed as an infra failure 0.0

The differing rewards are model nondeterminism on a 3-minute cap, not a property of the change. The before arm is the sharper illustration of the bug: the verifier scored it 1.0 and Harbor still filed it as AgentTimeoutError.

Not run

Full-budget (3600s) runs of the shipped profiles, and the --compare opencode arm, which this change does not touch.

Deferred

agentTimeoutMultiplier on the six maka-* profiles in terminal-bench-smoke-profiles.json is now reachable only through the malformed-budget fallback, so it reads as a live control that no longer controls anything. Deleting it is a manifest semantics change that also moves that fallback's phase from declared × 4 to declared × 1, which deserves its own decision rather than riding along with a timeout fix.

opencode still derives its phase from a multiplier, so --compare gives both arms the same model budget only on a 900s-declared task. Pre-existing, untouched here, and fixable by giving that profile an absolute budget of its own.

Harbor resolves the agent phase as
`min(override_timeout_sec ?? task_declared, max_timeout_sec ?? inf) * multiplier`
(harbor/trial/trial.py, _resolve_timeout_sec). The smoke harness published the
cell budget plus its settlement grace on max_timeout_sec, which is a ceiling: it
can only lower the task's declared timeout, never raise it. For maka-basic on the
default 900s task Harbor folded that to min(900, 3630) * 4 = 3600 — the cell
budget exactly — so the cell was SIGKILLed at the instant it stopped calling the
model and began writing maka-cell-output.json.

Publish the phase on override_timeout_sec instead, and normalise
agent_timeout_multiplier to 1.0 wherever an absolute phase is published: Harbor
applies the multiplier after the min(), and the multiplier's only job — stretching
a task-declared timeout into the budget the profile wants — is already absorbed by
the absolute value. That also drops a premise the old comment took for granted.
The profile multipliers were tuned against a 900s declared timeout alone, but the
dataset declares 600/900/1800/3600, so `declared * multiplier == cell budget` was
never true in general; the phase now tracks MAKA_CELL_TIMEOUT_SEC on every task.
Profiles with no cell budget of their own keep their multiplier untouched.

The tests now assert the phase Harbor actually resolves rather than a field value.
A tail on max_timeout_sec satisfies a field-shaped assertion while folding back to
the task's own timeout, which is how this stayed green while the window was
unreachable.
… hid

Both producers' suites had grown their own copy of Harbor's resolution rule
(`min(override ?? task_declared, max ?? inf) * multiplier`). Two hand-written
copies of one external system's rule is how this bug class survives: if Harbor
changes the rule, both suites stay green and both are wrong. Move it to
__tests__/helpers/harbor-agent-phase.ts and have both import it. The shared
version models the multiplier, which the fixed-prompt copy deliberately did not,
so that suite gets strictly more faithful assertions for free.

Close the gaps that copy left in the smoke suite:

- agent_timeout_multiplier: 1.0 was unpinned. Every deadline test ran against a
  manifest whose timeout_multiplier is 1.0, where 1.0 and null are
  indistinguishable — so regressing that line to null passed the whole suite
  while Harbor would rescale the absolute phase through the job-level knob. A
  manifest with timeoutMultiplier 0.5 makes the difference observable: null
  resolves 3630 * 0.5 = 1815, under the cell's own 3600s budget, which is the
  kill this fix exists to prevent.
- The declared-timeout sweep claimed to list the dataset's declared timeouts and
  did not. Independence from the declared timeout is the actual invariant, so
  bracket every shipped budget with one value below and one above instead of
  snapshotting a list that goes stale silently.
- Nothing tied the arms together: maka's phase and opencode's were asserted
  against separate literals, so changing a cell budget would leave --compare
  silently incomparable. Assert the shared model budget and the grace-sized
  difference, the way the fixed-prompt suite already does.
- The smoke path never pinned an operator-widened MAKA_CELL_SETTLEMENT_GRACE_SEC.
- The phase invariant covered 2 of 6 maka profiles; cover all six.

Also drop an assertion duplicating one already made in the oracle test.
@Astro-Han
Astro-Han force-pushed the fix/headless-smoke-settlement-window branch from f1045e8 to 9d3f58c Compare August 4, 2026 08:24
@Astro-Han
Astro-Han marked this pull request as ready for review August 4, 2026 08:34
@Astro-Han
Astro-Han merged commit 13cf3e6 into main Aug 4, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant