Skip to content

fix(headless): let an authoritative reward survive a missing agent self-report - #2112

Merged
Astro-Han merged 3 commits into
mainfrom
fix/headless-budget-exhausted-authoritative-reward
Aug 4, 2026
Merged

fix(headless): let an authoritative reward survive a missing agent self-report#2112
Astro-Han merged 3 commits into
mainfrom
fix/headless-budget-exhausted-authoritative-reward

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Harbor's single_step.py and Pier both run the verifier after any agent-phase exception, so a trial that ended on AgentTimeoutError can still carry an authoritative reward. Both runners nonetheless required maka-cell-output.json — the agent's own self-report — before a budget exhaustion could keep its grade:

if (rewardArtifact === null || verifierArtifact === null || cellArtifact === null) {
  throw new FixedPromptBudgetExhaustedError(...)
}

An agent that ran out of time before filing that report therefore had a verifier-confirmed reward = 1.0 recorded as passed = false, scored = false. Six Maka cells in deepseek-v4-flash-3arm-tbench-2.1-full-v7 (adaptive-rejection-sampler, caffe-cifar-10, largest-eigenval, mcmc-sampling-stan, path-tracing, schemelike-metacircular-eval) were dropped this way; the Codex arm's identically-shaped adaptive-rejection-sampler cell scored correctly only because its self-report is written by an adapter wrapper that always runs.

The root cause of the missing self-report is fixed in #2107 (Harbor's max_timeout_sec is a ceiling, not an override, so Maka's 30s settlement window did not mathematically exist). This PR fixes the grading defect that #2107 records as a Known gap: an authoritative verdict must not be vetoed by a non-authoritative artifact.

Approach. No synthetic HarborCellOutput — its required fields (runtimeRefs, steps, startedAt, finishedAt, toolSummary) would have to be invented, which is worse than the defect. Instead the authoritative grade travels along the budget-exhausted path:

  • FixedPromptBudgetExhaustedError.artifactRefs gains harbor: { reward, verifier }, read from the verifier's own artifacts (verifier/reward.txt + maka-verifier-outcome.json).
  • taskBudgetExhaustedEvent projects passed / scored from it, under the rule taskCompletedScoringProjection already uses for a settled deadline: a settled deadline counts as verifierGraded, and reward > 0 is a pass. FixedPromptTaskBudgetExhaustedEvent.passed/scored widen from false to boolean and the event carries harbor.
  • Evidence that fails to attest the arm (identity mismatch, requireExecutionIdentity with no identity) keeps the outcome unscored — a grade nobody can attribute is not this arm's grade.
  • A verifier that did not conclude (candidate_timeout, infra_failed) or artifacts that do not parse yield no grade, so those exhaustions stay exactly as they are today.

Only facts that exist are recorded: the agent did exhaust its budget, and the verifier did reach a verdict.

Pier carried the identical defect (settledByEvidence = grade.state === 'graded' && cellArtifact !== null) and is fixed the same way; its recovery contract is already declared as non-forking with Harbor's.

Refs #2107

Verification

  • npm run --workspace @maka/headless test — 1367 pass, 0 fail, 1 skipped
  • npm run lint, npm run format:check — clean
  • npm run typecheck (all workspaces) — clean

The decisive test drives the real createHarborTaskRunner through runFixedPromptController, so a trial directory on disk is projected into the WAL row the benchmark counts — the seam where the six cells were actually lost:

  • scores a timed-out trial the verifier graded, from trial dir to WALpassed/scored/eligible = true, harbor.reward = 1
  • refuses to score a timed-out trial whose reward disagrees with its verifier outcomepassed/scored = false
  • refuses to score a timed-out trial whose verifier never concludedpassed/scored = false

Plus a table covering the runner's own transport contract (reward 0, empty, non-numeric; missing or malformed verifier outcome), controller-level projection cases, and an A/B assertion on the paired outcome a graded timeout now decides — not just the arm-level rate.

Review round: the first cut had a real defect

External review (three independent passes) caught that the first commit re-derived "what counts as a grade" at the new seam instead of reusing the existing judge. It read the artifacts and scored on reward > 0 alone, skipping assertVerifierRewardAgreement — the check the completed path enforces. Reproduced end to end: a reward.txt of 1.0 alongside a verifier outcome of failed — artifacts the completed path rejects as HarborInfraError — was recorded as passed: true, scored: true. A corrupt scoring authority became a fabricated pass, worse than the drop this PR fixes.

Fixed at the owner in 459ae43: the runner now only transports what the verifier wrote, and the controller derives passed/scored through structuredVerifierGrade + taskCompletedScoringProjection — the same pair the completed path uses. Disagreeing artifacts and a non-conclusive verifier both fall out as ungraded, so the runner's duplicated conclusiveness guard and its second parse of the same string were deleted with them. One HarborTrialGrade type now spans trial dir → error → WAL.

Blast radius

ab-summary needed no change: abOutcomeCategory already classifies a task_budget_exhausted event with no evidenceErrorClass as budget, isEvaluatedOutcome already admits it to the pass@1 denominator, and summarizeArm/summarizePairedAttempts already sum event.passed. A task_completed cell with errorClass = 'budget_exhausted' (the path taken when the self-report does exist) has always been able to be passed: true there, so this only restores symmetry between the two shapes. The new test pins the paired consequence explicitly.

prompt-acceptance-policy.ts gates on task_completed and is unaffected.

Second review round: propagation

A fresh-eyes round (three independent passes, no prior context) could not refute the derivation itself — no input produces a fabricated pass. It found that carrying the verdict was only half the change: the verdict also had to survive the checks every other scored path runs, and mean the same thing to every consumer.

Provider outage bypass. The budget-exhausted throw happens before incompleteTerminalProviderRequest, whose own contract says an upstream-truncated tail request stays infra however the trial settled — "that is what keeps a provider outage out of the denominator instead of scoring it as the agent's zero". Pre-branch that bypass was harmless because the event was always a non-pass; this branch made it consequential. The grade is now withheld on that path through one shared predicate used by both runners: the agent did not get the run it was given, so its verdict is not this arm's evidence. Mutation-verified (removing the guard fails both new tests).

RSI round analysis. taskOutcome read every task_budget_exhausted as budget, and isCovered required task_completed. A graded timeout pass was therefore reported as a pass → budget flip and a coverage regression — charging the candidate prompt for evidence that exists and says the task passed, while the acceptance gate counted the same row as a pass. Coverage now asks the event's own eligible/scored instead of its type; ungraded exhaustions still read as budget.

Acceptance partition. summarizePromptAcceptancePartition has no type guard, so graded timeouts already counted correctly into passEligibleRate/coverageRate — the intended consequence of this fix, but untested. Now pinned, alongside the ungraded case it must keep excluding. (An earlier note in this PR said the acceptance policy was unaffected; that was true of isStableBaselineEvent and selectAddressablePromptTasks, not of the partition summary.)

Doc. The WAL comment claimed an authoritative reward exists whenever harbor is present. It does not — a rejected verdict travels too. scored is the authority.

Deferred, with reasons

  • agent_exit without a self-report stays task_infra_failed. It claims no deadline, so task_budget_exhausted would lie, and task_completed needs the runtimeRefs/steps only the cell attests. Widening it is a WAL taxonomy decision, not this fix. Now documented at the branch rather than left silent.
  • RSI attribution (rsi-round-analysis.ts, rsi-controller-attribution.ts) buckets every task_budget_exhausted as budget before reading passed. That bucket is a deliberate category — prompt attribution should not credit a prompt for a timeout — and the new events land in the category matching their name. Changing it is a methodology decision about prompt optimization, not part of this defect.
  • Torn/partial maka-cell-output.json on the budget path still falls to infra: a non-empty but unparseable file is not null, so it takes the settled path and readCellOutput throws. Pre-existing behavior, unchanged here; distinguishing "absent" from "unreadable" at the gate is a separate change.

…lf-report

Harbor and Pier run the verifier after any agent-phase exception, so a trial
that exhausted its budget can still carry an authoritative reward. Both runners
nonetheless required maka-cell-output.json — the agent's own self-report — before
a budget exhaustion could keep its grade, so an agent that ran out of time before
filing one had its verified pass recorded as passed=false, scored=false.

Carry the verifier's verdict on the budget-exhausted path instead of synthesizing
a self-report nobody wrote: FixedPromptBudgetExhaustedError now transports the
reward and structured verifier outcome, and task_budget_exhausted projects
passed/scored from them under the same rule task_completed already uses for a
settled deadline (a settled deadline is verifierGraded; reward > 0 is a pass).
Evidence that fails to attest the arm keeps the outcome unscored.

Only facts that exist are recorded: the agent did exhaust its budget, and the
verifier did reach a verdict.
…g judge

The first cut re-derived "what counts as a grade" at the new seam: it read the
reward and verifier artifacts, then scored on reward > 0 alone. That skipped the
agreement check the completed path enforces (assertVerifierRewardAgreement), so
a reward of 1 alongside a verifier outcome of failed — artifacts the completed
path rejects as infra — became passed = true in the WAL. A corrupt scoring
authority turned into a fabricated pass, which is worse than the drop it fixes.

Give the decision back to its owner. The runner now only transports what the
verifier wrote; the controller derives passed/scored through structuredVerifierGrade
and taskCompletedScoringProjection, the same pair the completed path uses.
Disagreeing artifacts and a verifier that never concluded both fall out as
ungraded, so the runner's duplicated conclusiveness guard and its second parse
of the same string are gone with them.

Also: one HarborTrialGrade type from the trial dir through to the WAL, and an
explicit note that the agent_exit branch's cell requirement is a tracked
exclusion, not the veto fixed here.

Tests: run the real Harbor runner through the controller so a trial directory on
disk is projected into the WAL row the benchmark counts — the seam where the six
cells were actually lost, and where the disagreement defect was caught. The
runner's own contract becomes a table covering reward 0, empty, non-numeric, and
missing or malformed verifier artifacts. A/B coverage asserts the paired outcome
a graded timeout now decides, not just the arm-level rate.
…eline's rules

Carrying the verifier's verdict past a missing self-report was only half the
change: the verdict now had to survive the checks every other scored path runs,
and mean the same thing to every consumer of the WAL. Three places did not
follow.

The budget-exhausted throw happens before the terminal provider-request check,
so a trial whose upstream cut the stream short skipped the one guard that keeps
a provider outage out of the denominator — and could now record a pass there.
Withhold the grade on that path: the agent did not get the run it was given, so
its verdict is not this arm's evidence. One shared predicate, both runners.

RSI round analysis read every task_budget_exhausted as 'budget' and required
task_completed to count as covered, so a graded timeout pass was reported as a
pass→budget flip and a coverage regression — charging the candidate prompt for
evidence that exists and says the task passed. Coverage now asks the event's own
eligible/scored rather than its type.

The acceptance partition already counted the new events correctly; that was
untested, and is now pinned alongside the ungraded case it must keep excluding.

Also: the WAL doc claimed harbor's presence implied an authoritative reward. It
does not — a rejected verdict travels too. scored is the authority.
@Astro-Han
Astro-Han marked this pull request as ready for review August 4, 2026 09:03
@Astro-Han
Astro-Han merged commit a04bcd1 into main Aug 4, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant