feat(inspector): project a per-session causal trace over both ledgers

[ARE404]

Summary

First layer of the Session Inspector: a pure per-session causal trace over the two ledgers that already record it. No UI, no I/O, no new store.

This is the PR that #1625 paused for. The RFC's headline — per-step latency and cost — was not derivable from RuntimeEvent: one aggregate usage event per send, carrying a runtimeSteps count of how many steps it stood for. Rather than ship a view that was knowingly incomplete on exactly the turns worth inspecting, the metering source was fixed first (#1687, #1755, #1877). ModelCallAttempt now records one canonical record per physical provider request — step, attempt, latencyMs, timeToFirstTokenMs, usage with a usageBasis, and a cost frozen at call time with the rates it used — so the projection can finally be written honestly.

projectSessionTrace joins:

• RuntimeEvent for causal structure — tool dispatch paired with the result that answers it, permission decisions, errors.
• ModelCallAttempt for metering — what each call cost and how long it took.

Contract in packages/core, builder in packages/runtime, pure and synchronous with both ledgers handed in already read. The caller owns the I/O, which keeps @maka/storage out of @maka/runtime and the projection testable against fixtures.

Refs #1625. Builds on #1679.

Three properties that are deliberate

Retries are nested, not flattened. Attempts of one logical call share a logicalCallId by contract, so "this call was retried twice" is a grouping. A flat list of three steps that happen to share an id makes the reader reconstruct it.

An absent price stays absent. A step whose attempts were never priced carries no costUsd, and a session of only such calls totals to no price rather than to zero. That distinction is the one the canonical record exists to keep: unpriced and free are different facts.

Coverage is stated, not implied. A backend outside canonical accounting produces a trace that says so. The pi backend is exactly this case — recordModelCallAttempt has zero references in pi-agent-backend.ts, it emits token_usage only — and an empty timeline is indistinguishable from a session that did nothing. Aggregate usage with no canonical record behind it is the signal, reported per session as absent or partial with the turns named.

Failure attribution points at what failed first, not at the terminal error. A turn that ends in an error usually ends there because of an earlier tool failure; naming the last event names the symptom.

One thing I'd like confirmed before it stays

TraceTotals.costUsd sums a session's attempts. @Astro-Han warned in the issue thread that a per-session cost would make the Inspector a second cost authority, and at the time that was right — Settings → Usage aggregated the telemetry index while the Inspector would have projected the RuntimeEvent ledger, two record paths over one pricing table.

That inverted with #1755: Settings → Usage now aggregates these same ModelCallAttempt records. Summing a session's attempts is reading one authority at a different scope, not creating a second — a disagreement between the two would be a bug in one of them rather than two defensible numbers. I've written it that way, but it's the claim I'm least willing to assert unilaterally. If you'd rather it wait, dropping costUsd from TraceTotals touches one field and one test; everything else stands.

Deliberately not here

No searchTrace API — structured predicates are caller-side Array.filter over TraceStep[], per the issue discussion. No UI; the fifth SessionWorkbar tab is PR 2. No pricing parameter; cost is read off the record as stored rather than recomputed from a current snapshot. Nothing touches Runtime Host ownership or the Desktop writer path.

Two coverage gaps are reported rather than closed: the pi backend above, and goal_evaluation, which still writes only the frozen table because it has no run or turn identity at the Host layer.

Verification

• 9 node tests covering retry grouping, unpriced totals, compaction attribution inside a turn, tool dispatch/result pairing, failure attribution across failed-tool → terminal error, the pi-shaped session, partial coverage, no-model-activity sessions, and partial streaming events.
• @maka/core 703/703, @maka/runtime 2721/2733 — the 3 failing suites are this machine's rg-as-a-shell-function sandbox noise, identical on the parent commit.
• npm run format:check and node scripts/check-console.mjs clean.

🤖 Generated with Claude Code

[ARE404]

test is red twice here, and I don't think it's this branch — but I'd rather lay out the evidence than assert it, since I said the same thing about an e2e red on #1877 and was only right because I checked.

Two runs of identical content failed different tests. Run 1: host-kernel.test.ts — "a Host launched by the public Electron Client survives its parent process". Run 2 (same commit, rebased onto 75d20cf7): goal-root-authority.test.ts — "Automation turns can use Goal tools and contribute evaluation evidence". Both are heavy lifecycle tests in @maka/runtime-host.

This diff is 891 added lines and touches no runtime-host code: a new contract file in packages/core, a new pure projection in packages/runtime, one export line, one export-map entry. There is no path from it to Electron process teardown or Goal authority timing.

Why it lands on this PR rather than on main. Touching packages/core makes ci-test-plan.mjs select every workspace:

workspaces=packages/core,packages/storage,packages/mcp,packages/runtime,packages/runtime-host,packages/computer-use,packages/headless,packages/cli,packages/ui,apps/desktop

So a core-touching PR runs the full matrix under parallel load, while a UI-only push on main never exercises these suites. #1934's own description says exactly this: "The full workspace CI run after #1928 exposed two independent existing timing assumptions under parallel load." That fix is already in my base (cd20185e), and it touched host-kernel.test.ts — just not the case run 1 hit.

The counter-evidence, for completeness: #1877 also touched packages/core and its full matrix passed about five hours ago. So this is either a set of races that widened with today's runtime-host work (#1917, #1926, #1928, #1946), or luck. I can't distinguish those from outside, and gh run rerun isn't available to me as an external contributor — a third push would be guessing rather than testing.

So: happy to wait for the de-racing to settle and rebase, or to look into either failure properly as its own change if that's useful. What I'd rather not do is keep pushing this branch to reroll the dice.

[ARE404]

Green after rebasing onto 2030460a: test 5m13s, typecheck, changes all pass (e2e/storybook skip — no UI in this PR).

The rebase picked up #1964, which split execution-host.test.ts into parallel files. That is consistent with the load/timing diagnosis rather than proof of it — same branch content, third run, different base — but the branch is green and the two failures were in suites this diff does not touch. Nothing here changed to make it pass.

[Astro-Han]

The overall shape looks good to me: the contract/builder split follows the existing package boundary, the projection stays pure, retries retain their logical grouping, and unpriced spend is not collapsed into a false $0. Approving this direction.

I found six correctness follow-ups that are worth addressing. I don't think they rise to P1 for this currently unconsumed read model, but they should be fixed before the Inspector UI relies on these fields:

1. Model-call coverage overstates what is known. A turn with tokenUsage.runtimeSteps = 2 and only one logical main-call attempt is currently reported as complete. The canonical ModelCallAttempt contract explicitly says present settlements cannot prove completeness. Detectable shortfalls should be partial, and the no-known-gap state should not imply a completeness proof. (packages/runtime/src/session-trace-projection.ts:55-94, packages/core/src/session-trace.ts:191-199)

2. Attempts need canonical attemptId deduplication. A provisional abort and its later settlement can be appended under the same attempt id. Without dedupeModelCallAttempts, the trace invents a retry/unpriced attempt and can double-count repeated priced settlements, disagreeing with Settings → Usage. (packages/runtime/src/session-trace-projection.ts:132-163,300-312)

3. Step-less turns produce non-finite bounds. Usage-only, text-only, and history-compaction text turns have no projected steps, so Math.min/max(...steps) yields Infinity/-Infinity; JSON serialization turns those into null. Turn bounds should come from the input ledger facts, not only visible steps. (packages/runtime/src/session-trace-projection.ts:97-123)

4. Handled tool failures currently mark a completed turn as failed. A failed tool result followed by model recovery and terminal status: 'completed' still gets failure: { code: 'tool_failed' }. Terminal RuntimeEvent status should decide whether the turn failed; failed steps should only locate the cause after that fact is established. (packages/runtime/src/session-trace-projection.ts:271-293)

5. The public compaction step is never emitted. TraceCompactionStep exists, but real history-compaction checkpoints are system text RuntimeEvents and projectEventSteps ignores them. A history_compact model-call attempt represents the summarizer request, not the written compaction boundary. (packages/core/src/session-trace.ts:28,111-119, packages/runtime/src/session-trace-projection.ts:195-261)

6. recoveryMode is a policy, not evidence that recovery occurred. The dispatch field is present on normal first executions too, while actual recovery results live in actions.toolRecovery. The trace should distinguish the declared recovery policy from a durable recovery decision, correlated by operationId. (packages/core/src/session-trace.ts:97-98, packages/runtime/src/session-trace-projection.ts:201-215)

The current tests are green, but regression cases for raw duplicate settlements, usage-only turns, successful recovery after a tool error, detectable runtimeSteps gaps, compaction boundaries, and recovery decisions would close these gaps cleanly.

[ARE404]

Thanks — all six were right, and two of them were the failure mode I keep having to relearn: contract surface that promised more than the code delivered. Fixed in 0a0219da.

1. Coverage claimed a proof it cannot have. complete is gone; the state is now no_known_gap — the absence of evidence of a gap, which is what present settlements can actually support. The runtimeSteps shortfall is detected too, reported per turn as turnsWithFewerModelCallsThanSteps and documented as a floor on what is missing rather than a count of it.

2. attemptId dedupe. You're right, and this one had teeth: dedupeModelCallAttempts already exists and model-call-usage-projection.ts uses it — I read that file and still didn't. The trace invented a retry and double-counted a priced settlement, so a session total would have disagreed with Settings → Usage over the very same records, which is precisely the property I argued for in the PR description. Grouping now goes through core's groupModelCallAttempts, which dedupes on the way in.

3. Non-finite bounds. Math.min(...[]) is Infinity and JSON renders it null. Bounds now come from the ledger facts the turn is made of, not from whichever steps happened to project. My own pi-backend test constructed exactly this turn and asserted only the coverage field, so the bug sat inside a passing test.

4. Handled tool failures. Fixed as you describe: the ledger's terminal status decides whether the turn failed, and a failed step only locates the cause once that is established. A tool error the model recovered from now leaves the step failed and the turn not.

5. The compaction step was declared and never emitted. Written checkpoints are now projected from the history-compact: events, and I've written the distinction into the contract: the boundary the next request replays from is not the summarizer call that produced its text — one is structure, the other is spend, and a turn can show either without the other.

6. recoveryMode. Renamed to recoveryPolicy, since every dispatch declares one. An actual recovered decision is joined from actions.toolRecovery by operationId — correlated rather than positional, because the decision is appended by the recovery writer and not by the dispatch it settles.

Seven regression tests, one per finding plus the no-known-gap state: duplicate settlements, runtimeSteps shortfall, usage-only turn bounds (including a JSON round-trip, since null was the observable symptom), recovery after a tool error, compaction boundaries, and policy-versus-decision.

@maka/core 703/703, @maka/runtime 2742/2757 — the 6 local failures are this machine's rg-as-a-shell-function and macOS path-semantics noise, and CI ran the same base green.

[Astro-Han]

Re-reviewed the latest head. The six previous correctness issues are fixed with targeted regression coverage, and the current CI is green. Approving the PR.

One non-blocking P2 follow-up: failure attribution currently sorts projected steps by startedAt and then selects the first failed step. Because tool calls within a model step execute concurrently, this can attribute the turn to the tool that started first rather than the one that failed first. For example, tool A may start before B but finish with an error after B; the trace will point to A even though B failed first.

Please consider comparing failure timestamps instead: tool endedAt, the terminal model attempt completedAt, and error startedAt. A regression test with two concurrent tools completing in reverse order would lock the intended points-at-what-failed-first contract.