fix(headless): let an authoritative reward survive a missing agent self-report - #2112
Merged
Astro-Han merged 3 commits intoAug 4, 2026
Merged
Conversation
…lf-report Harbor and Pier run the verifier after any agent-phase exception, so a trial that exhausted its budget can still carry an authoritative reward. Both runners nonetheless required maka-cell-output.json — the agent's own self-report — before a budget exhaustion could keep its grade, so an agent that ran out of time before filing one had its verified pass recorded as passed=false, scored=false. Carry the verifier's verdict on the budget-exhausted path instead of synthesizing a self-report nobody wrote: FixedPromptBudgetExhaustedError now transports the reward and structured verifier outcome, and task_budget_exhausted projects passed/scored from them under the same rule task_completed already uses for a settled deadline (a settled deadline is verifierGraded; reward > 0 is a pass). Evidence that fails to attest the arm keeps the outcome unscored. Only facts that exist are recorded: the agent did exhaust its budget, and the verifier did reach a verdict.
…g judge The first cut re-derived "what counts as a grade" at the new seam: it read the reward and verifier artifacts, then scored on reward > 0 alone. That skipped the agreement check the completed path enforces (assertVerifierRewardAgreement), so a reward of 1 alongside a verifier outcome of failed — artifacts the completed path rejects as infra — became passed = true in the WAL. A corrupt scoring authority turned into a fabricated pass, which is worse than the drop it fixes. Give the decision back to its owner. The runner now only transports what the verifier wrote; the controller derives passed/scored through structuredVerifierGrade and taskCompletedScoringProjection, the same pair the completed path uses. Disagreeing artifacts and a verifier that never concluded both fall out as ungraded, so the runner's duplicated conclusiveness guard and its second parse of the same string are gone with them. Also: one HarborTrialGrade type from the trial dir through to the WAL, and an explicit note that the agent_exit branch's cell requirement is a tracked exclusion, not the veto fixed here. Tests: run the real Harbor runner through the controller so a trial directory on disk is projected into the WAL row the benchmark counts — the seam where the six cells were actually lost, and where the disagreement defect was caught. The runner's own contract becomes a table covering reward 0, empty, non-numeric, and missing or malformed verifier artifacts. A/B coverage asserts the paired outcome a graded timeout now decides, not just the arm-level rate.
…eline's rules Carrying the verifier's verdict past a missing self-report was only half the change: the verdict now had to survive the checks every other scored path runs, and mean the same thing to every consumer of the WAL. Three places did not follow. The budget-exhausted throw happens before the terminal provider-request check, so a trial whose upstream cut the stream short skipped the one guard that keeps a provider outage out of the denominator — and could now record a pass there. Withhold the grade on that path: the agent did not get the run it was given, so its verdict is not this arm's evidence. One shared predicate, both runners. RSI round analysis read every task_budget_exhausted as 'budget' and required task_completed to count as covered, so a graded timeout pass was reported as a pass→budget flip and a coverage regression — charging the candidate prompt for evidence that exists and says the task passed. Coverage now asks the event's own eligible/scored rather than its type. The acceptance partition already counted the new events correctly; that was untested, and is now pinned alongside the ungraded case it must keep excluding. Also: the WAL doc claimed harbor's presence implied an authoritative reward. It does not — a rejected verdict travels too. scored is the authority.
Astro-Han
marked this pull request as ready for review
August 4, 2026 09:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Harbor's
single_step.pyand Pier both run the verifier after any agent-phase exception, so a trial that ended onAgentTimeoutErrorcan still carry an authoritative reward. Both runners nonetheless requiredmaka-cell-output.json— the agent's own self-report — before a budget exhaustion could keep its grade:An agent that ran out of time before filing that report therefore had a verifier-confirmed
reward = 1.0recorded aspassed = false, scored = false. Six Maka cells indeepseek-v4-flash-3arm-tbench-2.1-full-v7(adaptive-rejection-sampler,caffe-cifar-10,largest-eigenval,mcmc-sampling-stan,path-tracing,schemelike-metacircular-eval) were dropped this way; the Codex arm's identically-shapedadaptive-rejection-samplercell scored correctly only because its self-report is written by an adapter wrapper that always runs.The root cause of the missing self-report is fixed in #2107 (Harbor's
max_timeout_secis a ceiling, not an override, so Maka's 30s settlement window did not mathematically exist). This PR fixes the grading defect that #2107 records as a Known gap: an authoritative verdict must not be vetoed by a non-authoritative artifact.Approach. No synthetic
HarborCellOutput— its required fields (runtimeRefs,steps,startedAt,finishedAt,toolSummary) would have to be invented, which is worse than the defect. Instead the authoritative grade travels along the budget-exhausted path:FixedPromptBudgetExhaustedError.artifactRefsgainsharbor: { reward, verifier }, read from the verifier's own artifacts (verifier/reward.txt+maka-verifier-outcome.json).taskBudgetExhaustedEventprojectspassed/scoredfrom it, under the ruletaskCompletedScoringProjectionalready uses for a settled deadline: a settled deadline counts asverifierGraded, andreward > 0is a pass.FixedPromptTaskBudgetExhaustedEvent.passed/scoredwiden fromfalsetobooleanand the event carriesharbor.requireExecutionIdentitywith no identity) keeps the outcome unscored — a grade nobody can attribute is not this arm's grade.candidate_timeout,infra_failed) or artifacts that do not parse yield no grade, so those exhaustions stay exactly as they are today.Only facts that exist are recorded: the agent did exhaust its budget, and the verifier did reach a verdict.
Pier carried the identical defect (
settledByEvidence = grade.state === 'graded' && cellArtifact !== null) and is fixed the same way; its recovery contract is already declared as non-forking with Harbor's.Refs #2107
Verification
npm run --workspace @maka/headless test— 1367 pass, 0 fail, 1 skippednpm run lint,npm run format:check— cleannpm run typecheck(all workspaces) — cleanThe decisive test drives the real
createHarborTaskRunnerthroughrunFixedPromptController, so a trial directory on disk is projected into the WAL row the benchmark counts — the seam where the six cells were actually lost:scores a timed-out trial the verifier graded, from trial dir to WAL→passed/scored/eligible = true,harbor.reward = 1refuses to score a timed-out trial whose reward disagrees with its verifier outcome→passed/scored = falserefuses to score a timed-out trial whose verifier never concluded→passed/scored = falsePlus a table covering the runner's own transport contract (reward
0, empty, non-numeric; missing or malformed verifier outcome), controller-level projection cases, and an A/B assertion on the paired outcome a graded timeout now decides — not just the arm-level rate.Review round: the first cut had a real defect
External review (three independent passes) caught that the first commit re-derived "what counts as a grade" at the new seam instead of reusing the existing judge. It read the artifacts and scored on
reward > 0alone, skippingassertVerifierRewardAgreement— the check the completed path enforces. Reproduced end to end: areward.txtof1.0alongside a verifier outcome offailed— artifacts the completed path rejects asHarborInfraError— was recorded aspassed: true, scored: true. A corrupt scoring authority became a fabricated pass, worse than the drop this PR fixes.Fixed at the owner in 459ae43: the runner now only transports what the verifier wrote, and the controller derives
passed/scoredthroughstructuredVerifierGrade+taskCompletedScoringProjection— the same pair the completed path uses. Disagreeing artifacts and a non-conclusive verifier both fall out as ungraded, so the runner's duplicated conclusiveness guard and its second parse of the same string were deleted with them. OneHarborTrialGradetype now spans trial dir → error → WAL.Blast radius
ab-summaryneeded no change:abOutcomeCategoryalready classifies atask_budget_exhaustedevent with noevidenceErrorClassasbudget,isEvaluatedOutcomealready admits it to the pass@1 denominator, andsummarizeArm/summarizePairedAttemptsalready sumevent.passed. Atask_completedcell witherrorClass = 'budget_exhausted'(the path taken when the self-report does exist) has always been able to bepassed: truethere, so this only restores symmetry between the two shapes. The new test pins the paired consequence explicitly.prompt-acceptance-policy.tsgates ontask_completedand is unaffected.Second review round: propagation
A fresh-eyes round (three independent passes, no prior context) could not refute the derivation itself — no input produces a fabricated pass. It found that carrying the verdict was only half the change: the verdict also had to survive the checks every other scored path runs, and mean the same thing to every consumer.
Provider outage bypass. The budget-exhausted throw happens before
incompleteTerminalProviderRequest, whose own contract says an upstream-truncated tail request stays infra however the trial settled — "that is what keeps a provider outage out of the denominator instead of scoring it as the agent's zero". Pre-branch that bypass was harmless because the event was always a non-pass; this branch made it consequential. The grade is now withheld on that path through one shared predicate used by both runners: the agent did not get the run it was given, so its verdict is not this arm's evidence. Mutation-verified (removing the guard fails both new tests).RSI round analysis.
taskOutcomeread everytask_budget_exhaustedasbudget, andisCoveredrequiredtask_completed. A graded timeout pass was therefore reported as apass → budgetflip and a coverage regression — charging the candidate prompt for evidence that exists and says the task passed, while the acceptance gate counted the same row as a pass. Coverage now asks the event's owneligible/scoredinstead of its type; ungraded exhaustions still read asbudget.Acceptance partition.
summarizePromptAcceptancePartitionhas no type guard, so graded timeouts already counted correctly intopassEligibleRate/coverageRate— the intended consequence of this fix, but untested. Now pinned, alongside the ungraded case it must keep excluding. (An earlier note in this PR said the acceptance policy was unaffected; that was true ofisStableBaselineEventandselectAddressablePromptTasks, not of the partition summary.)Doc. The WAL comment claimed an authoritative reward exists whenever
harboris present. It does not — a rejected verdict travels too.scoredis the authority.Deferred, with reasons
agent_exitwithout a self-report staystask_infra_failed. It claims no deadline, sotask_budget_exhaustedwould lie, andtask_completedneeds theruntimeRefs/stepsonly the cell attests. Widening it is a WAL taxonomy decision, not this fix. Now documented at the branch rather than left silent.rsi-round-analysis.ts,rsi-controller-attribution.ts) buckets everytask_budget_exhaustedasbudgetbefore readingpassed. That bucket is a deliberate category — prompt attribution should not credit a prompt for a timeout — and the new events land in the category matching their name. Changing it is a methodology decision about prompt optimization, not part of this defect.maka-cell-output.jsonon the budget path still falls to infra: a non-empty but unparseable file is notnull, so it takes the settled path andreadCellOutputthrows. Pre-existing behavior, unchanged here; distinguishing "absent" from "unreadable" at the gate is a separate change.