feat(metering): route compaction model calls through the canonical seam - #1877
Conversation
|
Verified the routing is correct: no double metering, each summarizer call settles exactly once through the tracker, the main-send refactor onto P0: none. P1
P2
P3
Happy to file a follow-up for routing |
… it settles Review round 1 on apache#1877. The P1 and the P2s that were about this PR's own behaviour, plus every P3. **Metering no longer depends on a diagnostic.** `createProviderRequestTracker` returned undefined without `recordProviderRequestCapture`, so a deployment with capture off — a reachable config; both hosts wire it conditionally — silently stopped metering compaction, which the deleted `recordLlmCall` had metered unconditionally. Capture is now optional inside the tracker: `preparedCapture` is pure, so `requestHash`, `requestBytes`, and `segments` survive without a sink, and only the artifact join keys (`captureId`, `captureArtifactId`) go absent. The tracker is built when there is either sink to feed. The Host now wires the summarizer's tracking unconditionally too, and the Desktop wires it at all — it never had, so the accounting the backend computed for a Desktop history compaction was handed to a summarizer that had nowhere to settle it. **The "one place owns attaching a tracker" claim is now true rather than asserted.** `withProviderGenerateTracking` is shared by `generateCompactSummary` and `buildLlmHistorySummarizer`; `ProviderMiddlewareGenerateInput` is declared once. Also fixed: history_compact records carried no `contextWindow` while semantic_compact did. **Tests for what was only claimed:** - `usageBasis: 'missing'` — the branch had no test anywhere. A call the provider reported no usage for records `missing`, not zero tokens, and stays unpriced whatever the resolver would have said. - Metering with capture switched off, asserting the attempt still carries the locally-computed request shape and no artifact ids. - Cache attribution end-to-end: the semantic mock now ships a `raw` provider payload, so the assertion moved from "absent because this mock claims nothing" to `cacheReadInputTokens: 2` through the provider branch. The no-`raw` rule stays pinned at the unit level. - A dry-run (`validate_only`) semantic compaction really is a billed call: the summarizer runs to completion and only then is its block refused. Confirmed by the test, which is why it is worth pinning — a mode named "dry run" that bills is what a later reader would assume otherwise. - Mid-turn history compaction settles a canonical record end-to-end through the backend glue, with the real summarizer against a mock provider, asserting the live `runId` resolves — a stubbed resolver cannot show that. P3 cleanups: two dead imports, and the Usage IPC comment that still claimed the frozen table receives compaction calls. `@maka/runtime` 2684/2696 (3 suites = documented local `rg` noise), `@maka/runtime-host` 524/524, `@maka/desktop` 1101/1139 (38 = this worktree's missing Astryx peer dep, unchanged from the parent commit). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Thanks — fixed in P1 — P2 — metering gated on the capture seam. Correct, and it was a regression this PR introduced: both hosts wire capture conditionally, so "capture off, compaction on" is reachable, and P2 — the "one place" claim. Correct, and it was my commit message asserting a property the code did not have. P2 — cache attribution / dry-run / mid-turn glue test gaps. All three added. The dry-run one confirmed your reading: P2 — errorClass: I think this one is wrong in its premise. The main path does not set P2 — manual P3. Both dead imports gone ( Yes please on the Verification: |
|
One blocker here, plus a pricing issue that I think belongs in this PR. P1: manual compaction drops the accounting record. The manual path is active. Desktop
The old I think the clean fix is to put P2: a separate semantic summarizer model is still priced as the main model. The tracker gets With This was already wrong before this PR, but canonical records now preserve the conflicting model and rates together. |
…all as its own model Review round 2 on apache#1877. Both findings were correct, and the first was a live bug I had argued my way out of. **Manual compaction was unmetered, and my "nothing reaches this path" was wrong.** Desktop `sessions:compact` and CLI `/compact` both reach `RuntimeKernel.compactSession()`, which opens an `AgentRun` and then called `compactHistory` without its id. Outside `send()` there is no `currentRunId`, so the summarization settled nothing while the old `recordLlmCall` had metered it. `BackendCompactHistoryInput.runId` is now **required**, not optional: the failure mode is silence, and an optional field is one a caller can forget in exactly the situation that produced this bug. The kernel passes `run.runId`; the compaction threads it into the accounting identity for that one call. Seventeen test call sites had to name a run, which is the type doing its job. **A configured summarizer model was priced as the session model.** Cost resolution looked up `${providerType}:${this.input.modelId}` regardless of which model served the request, so with `MAKA_CONTEXT_SEMANTIC_COMPACT_MODEL` set we stored one model's id beside another model's `pricingRates` — precisely what recording the rates exists to prevent. `resolveModelCallCost` now takes the call's model id, supplied through the same identity the tracker already carries. **One bug the new test caught in my own fix:** the compaction dep was wired as `(callKind) => this.modelCallAccounting(callKind)`, which silently dropped the new identity argument. TypeScript accepts a narrower function, so the manual compaction kept recording nothing and the types stayed green. Only the end-to-end assertion showed it. Tests, both as asked: `SessionManager.compactSession()` settles exactly one `history_compact` record carrying the run the kernel opened, driven through a real `AiSdkBackend` and a real summarizer; and a semantic compaction with a distinct summarizer model records that model's own rates while the send's own steps keep the session model's. `@maka/runtime` 2686/2698 (3 suites = documented local `rg` noise), `@maka/runtime-host` 524/524, `@maka/desktop` 1101/1139 (38 = this worktree's missing Astryx peer dep). All runtime-consuming packages build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Both correct. Fixed in P1 — manual compaction drops the record. You're right, and I was wrong in the way that matters: I wrote "nothing reaches this path today" in the PR description without checking, after you'd flagged it the first time.
The test you asked for caught a second bug inside my own fix. The compaction dependency was wired as P2 — summarizer model priced as the main model. Also correct, and it belongs in this PR for the reason you gave: the record now preserves the id and the rates together, so the contradiction became storable. Verification: |
|
The red
Nothing in this PR touches the renderer or project grouping; |
First of the two compaction call kinds apache#1679 left unrouted. `history_compact` was still writing a per-send row into the frozen `LlmCallRecord` table, which is the last place a model call is metered outside `ModelCallAttempt`. The summarizer already built a `ProviderRequestTracker` for capture and attempt diagnostics; it now also carries accounting, so the call settles the same way a main send does — one record per physical provider request, with `usageBasis` and `costBasis` instead of a cost that cannot say whether it is real. **Accounting is supplied per call, by the backend.** The host wires the summarizer once at composition time and cannot know `runId`, which is per-turn state only the backend holds. Rather than plumbing a turn-to-run resolver down through the kernel and the backend factory context so the host could look up something the backend already has, `AiSdkBackend.modelCallAccounting(callKind)` builds the identity and the caller passes it in at the moment the call is made. The main send path now uses the same factory, so the three call kinds share one construction instead of repeating it. The legacy writer goes out in the same commit: keeping both would have made history compaction the double-metered path that apache#1755 removed everywhere else. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The second of the two compaction call kinds apache#1679 left unrouted, and the last model call anywhere that was metered by hand. `semantic_compact` had no tracker at all: it called `generateCompactSummary` directly and then built an `LlmCallRecord` around the returned usage. It now settles the way every other provider request does. `generateCompactSummary` takes an optional `ProviderRequestTracker` and wraps the model with a `wrapGenerate` middleware — the exact mirror of what `startStream` already does with `wrapStream`, and in the one place that already owns "attach a tracker to a model". The success, failure, and abort paths all settle inside the tracker, so the hand-rolled try/catch that recorded three ways goes with it. The backend hands the summarizer a *built* tracker rather than the capture, attempt, and id sinks it is made of: compaction has no business assembling metering identity, and a half-wired tracker is what produces records nothing can attribute. `createProviderRequestTracker` also absorbed the main send's own tracker construction, so this kind was added without a second copy of it. One trace per turn rather than per call, so a step that summarizes is a step of that trace and a retried summarization is another attempt of the same logical call. Built on first use — most turns never summarize, and an unused trace id is a trace that never happened. The legacy writer goes out in the same commit, along with the now-unused `computeCostUsd` dep: cost is resolved at settlement by the seam's own `resolveCost`, with a basis attached. One behavioural difference worth stating: the old row copied the SDK's normalized `cacheRead` through as a cache hit. The canonical record attributes cache tokens only when the provider's own payload claims them, so a provider that reports none now yields an absent field instead of a number no provider ever said. `@maka/runtime` 2648/2661 (the 4 failures are the documented local `rg` noise). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
With both compaction kinds routed, nothing in the send path writes a `LlmCallRecord` any more. `AiSdkCompactionCapabilities.recordLlmCall` and its `LlmTelemetryRecorder` type had no writer left above them, and the Host and Desktop each still wired a recorder into a backend field nothing read — dead plumbing that a future change could just as easily have brought back to life. Five backend tests still collected the records into arrays nothing asserted on; those go too. **`recordLlmCall` itself stays, and apache#1679's plan to delete it here was wrong.** `goal_evaluation` is the fourth call kind, and `createHostGoalEvaluator` still writes it to the frozen table. It cannot follow the compaction kinds through this seam as-is: a `ModelCallAttempt` is identified by `(sessionId, runId, turnId)`, and the Host evaluates a goal against a `sessionId` alone — there is no run or turn at that layer to attribute the call to. Giving it one is a design question for the RFC, not something to smuggle into a routing change. The practical consequence is worth stating plainly, because apache#1679 currently says otherwise: `provenance.legacyRecords` will *not* fall to zero as the old table ages out. Goal evaluations keep landing there, so the merged read path stays load-bearing until that kind is routed too. `@maka/runtime` 2648/2661 (4 = documented local `rg` noise), `@maka/runtime-host` 488/488, `@maka/desktop` 1090/1128 — identical to this worktree's baseline on the parent commit (the 38 are its missing Astryx peer dep, not this change). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… it settles Review round 1 on apache#1877. The P1 and the P2s that were about this PR's own behaviour, plus every P3. **Metering no longer depends on a diagnostic.** `createProviderRequestTracker` returned undefined without `recordProviderRequestCapture`, so a deployment with capture off — a reachable config; both hosts wire it conditionally — silently stopped metering compaction, which the deleted `recordLlmCall` had metered unconditionally. Capture is now optional inside the tracker: `preparedCapture` is pure, so `requestHash`, `requestBytes`, and `segments` survive without a sink, and only the artifact join keys (`captureId`, `captureArtifactId`) go absent. The tracker is built when there is either sink to feed. The Host now wires the summarizer's tracking unconditionally too, and the Desktop wires it at all — it never had, so the accounting the backend computed for a Desktop history compaction was handed to a summarizer that had nowhere to settle it. **The "one place owns attaching a tracker" claim is now true rather than asserted.** `withProviderGenerateTracking` is shared by `generateCompactSummary` and `buildLlmHistorySummarizer`; `ProviderMiddlewareGenerateInput` is declared once. Also fixed: history_compact records carried no `contextWindow` while semantic_compact did. **Tests for what was only claimed:** - `usageBasis: 'missing'` — the branch had no test anywhere. A call the provider reported no usage for records `missing`, not zero tokens, and stays unpriced whatever the resolver would have said. - Metering with capture switched off, asserting the attempt still carries the locally-computed request shape and no artifact ids. - Cache attribution end-to-end: the semantic mock now ships a `raw` provider payload, so the assertion moved from "absent because this mock claims nothing" to `cacheReadInputTokens: 2` through the provider branch. The no-`raw` rule stays pinned at the unit level. - A dry-run (`validate_only`) semantic compaction really is a billed call: the summarizer runs to completion and only then is its block refused. Confirmed by the test, which is why it is worth pinning — a mode named "dry run" that bills is what a later reader would assume otherwise. - Mid-turn history compaction settles a canonical record end-to-end through the backend glue, with the real summarizer against a mock provider, asserting the live `runId` resolves — a stubbed resolver cannot show that. P3 cleanups: two dead imports, and the Usage IPC comment that still claimed the frozen table receives compaction calls. `@maka/runtime` 2684/2696 (3 suites = documented local `rg` noise), `@maka/runtime-host` 524/524, `@maka/desktop` 1101/1139 (38 = this worktree's missing Astryx peer dep, unchanged from the parent commit). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fallout from making capture optional on `ProviderRequestAttemptRecord`: the headless provider-request trace analyzer joins attempts to captures by id, and the id is no longer guaranteed to be there. Behaviour is unchanged for every trace this tool actually reads. It analyses a capture ledger, so an attempt that cannot name a capture is incomplete by its own definition and fails with the same diagnostic it already produced — and the decoder above it already rejects such records as `invalid_attempt` before the join runs. Only the type needed narrowing. Caught by CI, not by me: I checked the blast radius of the optional fields in `core`, `storage`, `runtime-host`, and `desktop`, and did not grep `headless`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…all as its own model Review round 2 on apache#1877. Both findings were correct, and the first was a live bug I had argued my way out of. **Manual compaction was unmetered, and my "nothing reaches this path" was wrong.** Desktop `sessions:compact` and CLI `/compact` both reach `RuntimeKernel.compactSession()`, which opens an `AgentRun` and then called `compactHistory` without its id. Outside `send()` there is no `currentRunId`, so the summarization settled nothing while the old `recordLlmCall` had metered it. `BackendCompactHistoryInput.runId` is now **required**, not optional: the failure mode is silence, and an optional field is one a caller can forget in exactly the situation that produced this bug. The kernel passes `run.runId`; the compaction threads it into the accounting identity for that one call. Seventeen test call sites had to name a run, which is the type doing its job. **A configured summarizer model was priced as the session model.** Cost resolution looked up `${providerType}:${this.input.modelId}` regardless of which model served the request, so with `MAKA_CONTEXT_SEMANTIC_COMPACT_MODEL` set we stored one model's id beside another model's `pricingRates` — precisely what recording the rates exists to prevent. `resolveModelCallCost` now takes the call's model id, supplied through the same identity the tracker already carries. **One bug the new test caught in my own fix:** the compaction dep was wired as `(callKind) => this.modelCallAccounting(callKind)`, which silently dropped the new identity argument. TypeScript accepts a narrower function, so the manual compaction kept recording nothing and the types stayed green. Only the end-to-end assertion showed it. Tests, both as asked: `SessionManager.compactSession()` settles exactly one `history_compact` record carrying the run the kernel opened, driven through a real `AiSdkBackend` and a real summarizer; and a semantic compaction with a distinct summarizer model records that model's own rates while the send's own steps keep the session model's. `@maka/runtime` 2686/2698 (3 suites = documented local `rg` noise), `@maka/runtime-host` 524/524, `@maka/desktop` 1101/1139 (38 = this worktree's missing Astryx peer dep). All runtime-consuming packages build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
b5280ca to
bfdf958
Compare
|
I rechecked the latest commit. The two fixes are sound: manual compaction now carries the run opened by the kernel, and a separate semantic summarizer is priced as its own model. One live path is still missing the canonical record. The remaining gap
So sequenceDiagram
participant CLI
participant Kernel as RuntimeKernel
participant Backend as AiSdkBackend
participant Summary as History summarizer
participant Ledger as AgentRun ledger
CLI->>Kernel: /compact
Kernel->>Kernel: open AgentRun
Kernel->>Backend: compactHistory(runId, turnId)
Note over Backend: recordModelCallAttempt<br/>was not passed by CLI
Backend->>Backend: accounting = undefined
Backend->>Summary: summarize()
Note over Summary: providerRequestTracking<br/>was not configured
Summary->>Summary: untracked provider call
Summary-->>Backend: summary
Backend-->>Kernel: compact result
Note over Ledger: no history_compact<br/>ModelCallAttempt
The new That test proves the runtime path works when everything is wired. It does not exercise the CLI factory that users actually reach. I still see this as a blocker for the claim that CLI There is a similar, older gap in Harbor. Its controller exposes I think the split ownership is the real problemToday tracker ownership depends on the call kind: flowchart LR
subgraph Roots["Product composition"]
Desktop
Host["Runtime Host"]
CLI
Harbor
end
subgraph Calls["Model-call paths"]
Main["main<br/>Backend creates tracker"]
Semantic["semantic_compact<br/>Backend creates tracker"]
History["history_compact<br/>Host builds tracker inputs,<br/>Backend supplies accounting"]
end
Tracker["ProviderRequestTracker"]
Record["ModelCallAttempt"]
Desktop --> Main
Desktop --> Semantic
Desktop --> History
Host --> Main
Host --> Semantic
Host --> History
CLI --> Main
CLI --> History
Harbor --> Main
Harbor --> Semantic
Main --> Tracker
Semantic --> Tracker
History -.->|"only if that product<br/>wired every input"| Tracker
Tracker --> Record
This has now failed in several different ways: capture originally gated metering, Desktop history tracking was absent, manual compaction had no run identity, an intermediate callback dropped the new identity argument, and CLI still lacks both pieces of wiring. Another local CLI patch would close today's hole. It would not remove the state in which a new composition root can silently omit half of accounting. Would you be open to finishing the tracker ownership cleanup in this PR? The narrow version would be:
The resulting boundary is simpler: flowchart TD
Desktop
Host["Runtime Host"]
CLI
Harbor
Sink["Canonical sink<br/>plus optional capture diagnostics"]
Backend["AiSdkBackend"]
Factory["One tracker factory"]
Main["main"]
History["history_compact"]
Semantic["semantic_compact"]
Tracker["ProviderRequestTracker"]
Authority["AgentRun<br/>ModelCallAttempt"]
Usage["Usage projection"]
Desktop --> Sink
Host --> Sink
CLI --> Sink
Harbor --> Sink
Sink --> Backend
Backend --> Factory
Factory --> Main
Factory --> History
Factory --> Semantic
Main --> Tracker
History --> Tracker
Semantic --> Tracker
Tracker --> Authority
Authority --> Usage
This matches the revised #1679 design: every real provider request kind crosses the same tracker seam; capture remains optional; run and model identity belong to the call; the old compaction writer can be removed without leaving a product-specific hole. I am not suggesting a unified Runtime Host or a broader accounting rewrite. If you want to keep the diff smaller, passing the two missing fields in CLI and adding a real CLI composition test would fix the immediate blocker. My preference is the backend-owned history tracker now. This PR already owns that routing change, and the split has caused enough silent misses that making the invariant structural seems cheaper than debugging the next composition root. After that, I would stop. I do not see a reason to add another abstraction beyond one tracker factory and real composition tests. 中文意见我重新检查了最新提交。之前两个问题已经正确修复:手动压缩携带 Kernel 创建的真实 但真实 CLI 入口仍然没有 canonical record。
新增的 Harbor 也有类似的既有缺口。controller 已经提供 我认为反复遗漏的根因是 tracker ownership 分裂:
这个拆分已经先后导致 capture 依赖、Desktop 漏接、manual run identity 缺失、中间回调丢参数,以及现在的 CLI 漏接。 建议考虑在当前 PR 中做一次窄范围收口:
这不是要求统一 Runtime Host,也不是扩大 accounting 设计。 如果希望保持最小 diff,只补 CLI 缺失的两个字段并增加真实 CLI composition 测试,也能解决当前 blocker。但我更倾向在本 PR 内把 history tracker ownership 收回 Backend。这个 PR 本来就在修改这条路径,而当前拆分已经造成了多次静默遗漏。 完成这一点后就应该停。我不建议继续增加其他抽象。 |
… tracker Review round 3 on apache#1877. The blocker was real: the CLI composition never passed `ctx.recordModelCallAttempt` to `AiSdkBackend` and never configured the summarizer's tracking, so `/compact` got the new `runId` and still had nothing to settle into. Harbor had the same gap for `semantic_compact`. Both are fixed here, but the reviewer's structural point is the one worth acting on: this is the fifth way the same split has failed, so the split goes. `history_compact` now gets its tracker from the same `AiSdkBackend.createProviderRequestTracker` that `main` and `semantic_compact` already use, and `HistoryCompactSummaryInput` carries a *ready* tracker instead of an accounting identity. `BuildLlmHistorySummarizerOptions.providerRequestTracking` is gone, and with it the Host's and Desktop's hand-assembled copies of the tracker's inputs — including the `contextWindow` plumbing added one round ago, which the backend now supplies for free because it knows the model. What a product owes accounting is now exactly one thing: the canonical sink. Nothing else can be half-wired, because nothing else is a product's to wire. `AiSdkCompactionDeps.modelCallAccounting` went with it — one factory, one seam. **Composition tests, because the last round's test proved the wrong thing.** The `SessionManager.compactSession()` test supplied both dependencies itself, so it demonstrated that the runtime works when everything is wired, not that the CLI wires it. The two new tests build backends through the real CLI and Harbor factories and assert the sink reaches the caller's recorder. Both were confirmed to fail with their production wiring reverted. CLI's gap turned out to be wider than the review described: with no sink passed at all, `main` sends were unmetered too, and had been since the frozen table's writer was removed — CLI never wired `recordLlmCall` either, so this composition root has simply never been inside accounting. Left out deliberately, as agreed: `goal_evaluation` routing, a global `errorClass`, and anything Inspector-shaped. `@maka/runtime` 2700/2712 (3 suites = documented local `rg` noise), `@maka/runtime-host` 563/563, `maka-agent` (CLI) 467/467, `@maka/headless` harbor-cell 102/102, `@maka/desktop` 1156/1194 (38 = this worktree's missing Astryx peer dep). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Took the backend-owned version. The blocker was real, and wider than described. CLI passes no sink at all — On the ownership split — you're right, and the count is what convinced me. Capture gating metering, Desktop's missing tracking, manual compaction's missing run, the callback that dropped the identity argument, CLI's missing sink: five failures, all the same shape, all "a composition root supplied some of what accounting needs." A sixth patch would have been the wrong answer. So:
CLI and Harbor both pass it now. Your point about the last round's test landed. Verification: Agreed on where to stop: |
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for following this through. The latest revision resolves the remaining wiring and ownership gaps from the earlier reviews:
- tracker construction now has a single owner in the backend;
- main, semantic-compaction, and history-compaction calls share the same canonical accounting path;
- CLI and Harbor now forward the canonical sink;
- the added composition and runtime tests cover the previously missing paths.
I also did a fresh-eye re-review of the final diff and found no remaining P0–P3 issues. The current boundary is both simpler and more reliable: product roots provide the sink, while the backend owns the run/model/cost context needed to build the tracker.
CI is green. Looks good to merge.
First slice of apache#1625, and the one that was blocked: the RFC's headline — per-step latency and cost — was not derivable from `RuntimeEvent`, which is why this issue paused for the accounting work (apache#1687, apache#1755, apache#1877). That work landed, so `ModelCallAttempt` now carries one record per physical provider request with `step`, `attempt`, latency, time-to-first-token, usage basis, and a cost frozen at call time. The projection this issue always wanted is now writable honestly. Pure and synchronous: both ledgers are handed in already read, so the caller owns the I/O and `@maka/storage` stays out of `@maka/runtime`. Contract in `packages/core`, builder in `packages/runtime`, per the split agreed on the issue. Three properties are deliberate: - **Retries are nested, not flattened.** Attempts of one logical call share a `logicalCallId` by contract, so "this call was retried twice" is a grouping rather than something a reader reconstructs from four steps that happen to share an id. - **An absent price stays absent.** A step whose attempts were never priced has no `costUsd`, and a session of only such calls totals to no price rather than to zero — the distinction the canonical record exists to keep. - **Coverage is stated, not implied.** A backend that emits no canonical records produces a trace that says so. The pi backend is exactly this case: it emits `token_usage` and zero `ModelCallAttempt`, and an empty timeline would be indistinguishable from a session that did nothing. Aggregate usage with no record behind it is the signal, reported per session as `absent` or `partial` with the turns named. Failure attribution points at what failed *first*, not at the terminal error: a turn that ends in an error usually ends there because of an earlier tool failure, and naming the last event names the symptom. No UI, no search API, no cost authority of its own — a per-session total sums the same `ModelCallAttempt` records Settings → Usage aggregates, read at a different scope. `@maka/core` 703/703, `@maka/runtime` 2721/2733 (3 suites = documented local `rg` noise). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First slice of apache#1625, and the one that was blocked: the RFC's headline — per-step latency and cost — was not derivable from `RuntimeEvent`, which is why this issue paused for the accounting work (apache#1687, apache#1755, apache#1877). That work landed, so `ModelCallAttempt` now carries one record per physical provider request with `step`, `attempt`, latency, time-to-first-token, usage basis, and a cost frozen at call time. The projection this issue always wanted is now writable honestly. Pure and synchronous: both ledgers are handed in already read, so the caller owns the I/O and `@maka/storage` stays out of `@maka/runtime`. Contract in `packages/core`, builder in `packages/runtime`, per the split agreed on the issue. Three properties are deliberate: - **Retries are nested, not flattened.** Attempts of one logical call share a `logicalCallId` by contract, so "this call was retried twice" is a grouping rather than something a reader reconstructs from four steps that happen to share an id. - **An absent price stays absent.** A step whose attempts were never priced has no `costUsd`, and a session of only such calls totals to no price rather than to zero — the distinction the canonical record exists to keep. - **Coverage is stated, not implied.** A backend that emits no canonical records produces a trace that says so. The pi backend is exactly this case: it emits `token_usage` and zero `ModelCallAttempt`, and an empty timeline would be indistinguishable from a session that did nothing. Aggregate usage with no record behind it is the signal, reported per session as `absent` or `partial` with the turns named. Failure attribution points at what failed *first*, not at the terminal error: a turn that ends in an error usually ends there because of an earlier tool failure, and naming the last event names the symptom. No UI, no search API, no cost authority of its own — a per-session total sums the same `ModelCallAttempt` records Settings → Usage aggregates, read at a different scope. `@maka/core` 703/703, `@maka/runtime` 2721/2733 (3 suites = documented local `rg` noise). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…#1956) * feat(inspector): project a per-session causal trace over both ledgers First slice of #1625, and the one that was blocked: the RFC's headline — per-step latency and cost — was not derivable from `RuntimeEvent`, which is why this issue paused for the accounting work (#1687, #1755, #1877). That work landed, so `ModelCallAttempt` now carries one record per physical provider request with `step`, `attempt`, latency, time-to-first-token, usage basis, and a cost frozen at call time. The projection this issue always wanted is now writable honestly. Pure and synchronous: both ledgers are handed in already read, so the caller owns the I/O and `@maka/storage` stays out of `@maka/runtime`. Contract in `packages/core`, builder in `packages/runtime`, per the split agreed on the issue. Three properties are deliberate: - **Retries are nested, not flattened.** Attempts of one logical call share a `logicalCallId` by contract, so "this call was retried twice" is a grouping rather than something a reader reconstructs from four steps that happen to share an id. - **An absent price stays absent.** A step whose attempts were never priced has no `costUsd`, and a session of only such calls totals to no price rather than to zero — the distinction the canonical record exists to keep. - **Coverage is stated, not implied.** A backend that emits no canonical records produces a trace that says so. The pi backend is exactly this case: it emits `token_usage` and zero `ModelCallAttempt`, and an empty timeline would be indistinguishable from a session that did nothing. Aggregate usage with no record behind it is the signal, reported per session as `absent` or `partial` with the turns named. Failure attribution points at what failed *first*, not at the terminal error: a turn that ends in an error usually ends there because of an earlier tool failure, and naming the last event names the symptom. No UI, no search API, no cost authority of its own — a per-session total sums the same `ModelCallAttempt` records Settings → Usage aggregates, read at a different scope. `@maka/core` 703/703, `@maka/runtime` 2721/2733 (3 suites = documented local `rg` noise). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(inspector): correct six attribution and coverage gaps in the trace Review on #1956. All six were right; four are cases the tests never drove and two were contract surface that promised more than the code delivered. **Coverage claimed a proof it cannot have.** `complete` said every call settled; the canonical contract says present settlements cannot prove completeness. The state is now `no_known_gap` — the absence of evidence of a gap. A detectable shortfall is also new: `tokenUsage.runtimeSteps` states how many tool-loop steps one aggregate stands for, and fewer main calls on record than that is a disagreement between the two ledgers, reported per turn as a floor on what is missing rather than a count of it. **Attempts are deduplicated by `attemptId`.** An aborted attempt and its later settlement are appended under one id; the ledger dedupes on write, a stream read does not. Without `dedupeModelCallAttempts` the trace invented a retry and double-counted a priced settlement, which would have put a session total out of step with Settings → Usage over the very same records. Grouping now goes through core's `groupModelCallAttempts`, which dedupes on the way in. **Step-less turns had non-finite bounds.** Usage-only and text-only turns project no steps, and folding an empty list gives ±Infinity — which JSON renders as `null`. Bounds now come from the ledger facts the turn is made of. **A handled tool failure no longer fails the turn.** Any failed step marked the whole turn failed, including a tool error the model recovered from before finishing normally. The ledger's terminal status decides whether the turn failed; the failed step only locates the cause once that is established. **The compaction step is emitted rather than merely declared.** `TraceCompactionStep` existed and nothing produced it. Written checkpoints are system text events, and they are a different fact from the `history_compact` model call: one is the boundary the next request replays from, the other is the spend. Both now appear. **`recoveryMode` was a policy wearing the name of an outcome.** Every dispatch declares one, including ordinary first executions. It is now `recoveryPolicy`, and an actual `recovered` decision is joined from `actions.toolRecovery` by `operationId` — correlated rather than positional, because the decision is appended by the recovery writer and not by the dispatch it settles. Seven regression tests, one per finding plus the no-known-gap state. `@maka/core` 703/703, `@maka/runtime` 2742/2757 — the 6 local failures are this machine's `rg`-as-a-shell-function and macOS path noise; CI ran the same base green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Summary
Routes the last two model-call kinds that #1679 left outside the canonical seam. After #1755 every main send settles into a
ModelCallAttempt, buthistory_compactandsemantic_compactwere still metered by hand into the frozenLlmCallRecordtable — so compaction was the one place where usage was recorded twice as two different shapes, and where a cost that could not say whether it was real landed as a number.Both now settle the way a main send does: one canonical record per physical provider request, carrying
usageBasisandcostBasis.Accounting is supplied per call, by the backend. The Host wires the summarizers once at composition time and cannot know
runId— that is per-turn state onlyAiSdkBackendholds. Rather than plumbing a turn-to-run resolver down through the kernel andBackendFactoryContextso the Host could look up something the backend already has,AiSdkBackend.modelCallAccounting(callKind)builds the identity and the caller passes it in at the moment the call is made. Adding kinds this way removed code: the main send path now uses the same factory it does.history_compact(6235a9a9) — the summarizer already built aProviderRequestTrackerfor capture and attempt diagnostics;HistoryCompactSummaryInputgained an optionalaccountingthatAiSdkCompactionsupplies at both call sites.semantic_compact(f33a4e4a) — this one had no tracker at all.ModelAdapter.generateCompactSummarynow takes an optional tracker and wraps the model with awrapGeneratemiddleware callingtrackGenerate, the exact mirror of whatstartStreamalready does withwrapStream— in the one place that already owns "attach a tracker to a model". The backend hands compaction a built tracker rather than the capture, attempt, and id sinks it is made of: a half-wired tracker is what produces records nothing can attribute. Success, failure, and abort all settle inside the tracker, so the hand-rolled try/catch that recorded three ways is gone.Refs #1679.
Correction to #1679:
recordLlmCallcannot be deleted yetThe RFC says that once both compaction kinds are routed, the send-level
recordLlmCallhas no writers left andprovenance.legacyRecordsfalls to zero as the old table ages out of the queried range. Both statements are wrong, and I would rather say so here than have the issue keep asserting it.goal_evaluationis the fourthModelCallKind, andcreateHostGoalEvaluatorstill writes it to the frozen table. It cannot follow the compaction kinds through this seam as-is: aModelCallAttemptis identified by(sessionId, runId, turnId), and the Host evaluates a goal against asessionIdalone — there is no run or turn at that layer to attribute the call to. Giving it one is a design question for the RFC, not something to smuggle into a routing change.So this PR deletes only what genuinely has no writer left:
AiSdkCompactionCapabilities.recordLlmCalland itsLlmTelemetryRecordertype, the Host and Desktop wiring into that backend field (both were feeding a field nothing read any more), and five backend tests that collected records nothing asserted on. TherecordLlmCallhelper, its exports, andTelemetryIndexWriter.recordLlmCallstay for the goal evaluator.Practical consequence for reviewers of the read path: the merged read path stays load-bearing.
legacyRecordswill not reach zero while goal evaluations keep landing inusage_llm_calls. Happy to open a follow-up issue for routinggoal_evaluationif you agree that's the right next slice.Known gaps, stated rather than hidden
Two things this PR deliberately does not do, both surfaced in review:
errorClassis not carried on canonical records. The oldsemantic_compactrow had one. No canonical record has one — for any kind, includingmain, whoseerrorClassgoes to thesend_diagnostics_recordedrun-trace event rather than toModelCallAttempt. The field is declared in the contract and populated by nobody. Implementing it means classifying at the tracker for all four kinds, which is a [RFC] Canonical model-call accounting: one ModelCallAttempt record per real provider request #1679 change, not a routing one.ManualFixed incompactHistorysettles nothing.b5280caa— and the claim that nothing reaches that path was wrong. Desktopsessions:compactand CLI/compactboth reachRuntimeKernel.compactSession().BackendCompactHistoryInput.runIdis now required (silence is the failure mode, so an optional field would be one a caller can forget), the kernel passesrun.runId, and aSessionManager.compactSession()test pins exactly onehistory_compactrecord carrying it.Behaviour change worth flagging
The old
semantic_compactrow copied the AI SDK's normalizedcacheReadthrough as a cache hit. The canonical record attributes cache tokens only when the provider's own payload claims them (strictProviderRequestUsage), so a provider that reports no native cache fields now yields an absent field rather than a number no provider ever said. This is the same rule every other kind has followed since #1687; compaction was the outlier.Verification
@maka/runtime2678/2691 — the 4 failures are this machine'srg-is-a-shell-function sandbox noise, identical on the parent commit.@maka/runtime-host524/524.@maka/desktop1090/1128 — the 38 failures are this worktree's missing Astryx peer dep; verified identical on the parent commit by stashing the change and re-running, so they are not from this PR.npm run format:checkandnode scripts/check-console.mjsclean.@maka/runtimeand@maka/runtime-hostafter rebuilding@maka/core/@maka/storage/@maka/runtimedists onto the rebased base.c61550d7; the one overlapping file with upstream (apps/desktop/src/main/session-stream.ts) merged without conflict.Test evidence for the routing itself is behavioural, not incidental: the
history_compactsummarizer test and the backend's semantic-compact test both now decode the canonicalModelCallAttemptand assert run attribution,usageBasis, andcostBasis— the old assertions on the frozen table's row shape are gone rather than kept alongside.🤖 Generated with Claude Code