fix(headless): make the smoke harness reach Maka's settlement window - #2114
Merged
Conversation
Harbor resolves the agent phase as `min(override_timeout_sec ?? task_declared, max_timeout_sec ?? inf) * multiplier` (harbor/trial/trial.py, _resolve_timeout_sec). The smoke harness published the cell budget plus its settlement grace on max_timeout_sec, which is a ceiling: it can only lower the task's declared timeout, never raise it. For maka-basic on the default 900s task Harbor folded that to min(900, 3630) * 4 = 3600 — the cell budget exactly — so the cell was SIGKILLed at the instant it stopped calling the model and began writing maka-cell-output.json. Publish the phase on override_timeout_sec instead, and normalise agent_timeout_multiplier to 1.0 wherever an absolute phase is published: Harbor applies the multiplier after the min(), and the multiplier's only job — stretching a task-declared timeout into the budget the profile wants — is already absorbed by the absolute value. That also drops a premise the old comment took for granted. The profile multipliers were tuned against a 900s declared timeout alone, but the dataset declares 600/900/1800/3600, so `declared * multiplier == cell budget` was never true in general; the phase now tracks MAKA_CELL_TIMEOUT_SEC on every task. Profiles with no cell budget of their own keep their multiplier untouched. The tests now assert the phase Harbor actually resolves rather than a field value. A tail on max_timeout_sec satisfies a field-shaped assertion while folding back to the task's own timeout, which is how this stayed green while the window was unreachable.
… hid Both producers' suites had grown their own copy of Harbor's resolution rule (`min(override ?? task_declared, max ?? inf) * multiplier`). Two hand-written copies of one external system's rule is how this bug class survives: if Harbor changes the rule, both suites stay green and both are wrong. Move it to __tests__/helpers/harbor-agent-phase.ts and have both import it. The shared version models the multiplier, which the fixed-prompt copy deliberately did not, so that suite gets strictly more faithful assertions for free. Close the gaps that copy left in the smoke suite: - agent_timeout_multiplier: 1.0 was unpinned. Every deadline test ran against a manifest whose timeout_multiplier is 1.0, where 1.0 and null are indistinguishable — so regressing that line to null passed the whole suite while Harbor would rescale the absolute phase through the job-level knob. A manifest with timeoutMultiplier 0.5 makes the difference observable: null resolves 3630 * 0.5 = 1815, under the cell's own 3600s budget, which is the kill this fix exists to prevent. - The declared-timeout sweep claimed to list the dataset's declared timeouts and did not. Independence from the declared timeout is the actual invariant, so bracket every shipped budget with one value below and one above instead of snapshotting a list that goes stale silently. - Nothing tied the arms together: maka's phase and opencode's were asserted against separate literals, so changing a cell budget would leave --compare silently incomparable. Assert the shared model budget and the grace-sized difference, the way the fixed-prompt suite already does. - The smoke path never pinned an operator-widened MAKA_CELL_SETTLEMENT_GRACE_SEC. - The phase invariant covered 2 of 6 maka profiles; cover all six. Also drop an assertion duplicating one already made in the oracle test.
Astro-Han
force-pushed
the
fix/headless-smoke-settlement-window
branch
from
August 4, 2026 08:24
f1045e8 to
9d3f58c
Compare
Astro-Han
marked this pull request as ready for review
August 4, 2026 08:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The smoke harness published the Maka cell budget plus its settlement grace on
max_timeout_sec. Harbor resolves the agent phase asmin(override_timeout_sec ?? task_declared, max_timeout_sec ?? inf) * multiplier(harbor/trial/trial.py,_resolve_timeout_sec), somax_timeout_secis a ceiling — it can only lower the task's declared timeout, never raise it. Formaka-basicon the default 900s task that folded tomin(900, 3630) * 4 = 3600, the cell budget exactly, and the cell was SIGKILLed at the instant it stopped calling the model and began writingmaka-cell-output.json. The symptompackages/headless/src/maka-settlement.tsdescribes.#2107 fixed the same bug on the fixed-prompt batch path and deliberately left this one out, because the two paths do not resolve the phase the same way:
harbor-task-runner.tssets noagent_timeout_multiplier, while the smoke profiles do (maka-heavyis 8). Harbor applies that multiplier after themin(), so moving an absolute value ontooverride_timeout_secalone would have it multiplied again — 7230 × 8 = 57840.So this publishes the phase on
override_timeout_secand normalisesagent_timeout_multiplierto 1.0 wherever an absolute phase is published. The multiplier's only job is stretching a task-declared timeout into the budget the profile wants, and an absolute value has already absorbed it. 1.0 rather than null: null makes Harbor fall back to the job-leveltimeout_multiplier, which would rescale the number we just published. Profiles with no cell budget of their own (opencode,oracle) keep their multiplier untouched.That also drops a premise the old comment took for granted. It claimed the profiles tune the multiplier so that
declared * multiplier == cell budget. The multipliers were tuned against a 900s declared timeout alone, but the dataset declares 600/900/1800/3600 (52/92 tasks at 900), so the identity was never true in general — on a 3600s taskmaka-heavywas resolving a 28800s phase against a 7200s budget. The phase now tracksMAKA_CELL_TIMEOUT_SECon every task, which is whatmaka-settlement.tssays the one rule is.Refs #2107.
Second commit: one model of Harbor's rule, not two
#2107 landed a
harborAgentPhaseSechelper inharbor-task-runner.test.ts; this PR had grown its own. Two hand-written copies of one external system's rule is how this bug class survives — if Harbor changes_resolve_timeout_sec, both suites stay green and both are wrong. It now lives once inpackages/headless/src/__tests__/helpers/harbor-agent-phase.ts. The shared version models the multiplier, which the fixed-prompt copy deliberately did not, so that suite gets strictly more faithful assertions.Verification
npm test --workspace=packages/headless(1353 tests, 0 fail),npm run lint,npm run format:check.Assertions go through the phase Harbor actually resolves, not a field value — a tail on
max_timeout_secsatisfies a field-shaped assertion while folding straight back to the task's own timeout, which is how this stayed green while the window was unreachable. Reverting onlyharbor-smoke-config.tsfails the regression tests and leaves the rest green. Regressingagent_timeout_multiplierfrom1.0tonull— invisible to every test before the second commit — now fails two.Live smoke run
Unit tests cannot show a SIGKILL, so both geometries were run through the real harness (Harbor + Docker + DeepSeek,
*sqlite-with-gcov). The shippedmaka-basicgeometry is budget 3600 / declared 900 / ×4 → phase 3600 = budget. Reproduced at 1/20 the wall clock as budget 180 / declared 900 / ×6⁄7 → phase 180 = budget; the "after" arm is the generated config unmodified.max_timeout_sec: 210,override: null, mult 6⁄7override_timeout_sec: 210,max: null, mult 1.0budget_exhausted(rc 124)maka-harbor.status.json"status": "running"budget_exhaustedmaka-harbor.stdout.jsonsettledByDeadline: true,taxonomy: budget_exhausted,benchmarkFailureShouldThrow: falseAgentTimeoutError,n_errored_trials: 1n_errors: 0The differing rewards are model nondeterminism on a 3-minute cap, not a property of the change. The before arm is the sharper illustration of the bug: the verifier scored it 1.0 and Harbor still filed it as
AgentTimeoutError.Not run
Full-budget (3600s) runs of the shipped profiles, and the
--compareopencode arm, which this change does not touch.Deferred
agentTimeoutMultiplieron the sixmaka-*profiles interminal-bench-smoke-profiles.jsonis now reachable only through the malformed-budget fallback, so it reads as a live control that no longer controls anything. Deleting it is a manifest semantics change that also moves that fallback's phase fromdeclared × 4todeclared × 1, which deserves its own decision rather than riding along with a timeout fix.opencodestill derives its phase from a multiplier, so--comparegives both arms the same model budget only on a 900s-declared task. Pre-existing, untouched here, and fixable by giving that profile an absolute budget of its own.