📊 feat: Real-Time Context Window & Token Usage Tracking#13670
Conversation
|
@codex review |
There was a problem hiding this comment.
Pull request overview
Adds end-to-end, branch-aware context window + token usage (and optional cost) tracking by streaming backend-calculated snapshots/usage over SSE, persisting them for resumable streams, and rendering a compact gauge + breakdown UI beside the chat input.
Changes:
- Stream and persist per-call token usage (
on_token_usage) and per-call context snapshots (on_context_usage) for live display + resume. - Add server-resolved model token config endpoint (
GET /api/endpoints/token-config) with optional pricing gated by interface config. - Implement client-side token index + per-conversation Jotai state, and a new input-adjacent gauge UI with breakdown and tests/e2e coverage.
Reviewed changes
Copilot reviewed 44 out of 44 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/data-schemas/src/methods/index.ts | Re-export createTxMethods from methods barrel. |
| packages/data-schemas/src/index.ts | Expose createTxMethods at package root. |
| packages/data-provider/src/types/runs.ts | Add UsageEvents + usage/context event payload types. |
| packages/data-provider/src/types/agents.ts | Extend run/resume types with collectedUsage + contextUsage. |
| packages/data-provider/src/types.ts | Add tokenomics + token-config map types. |
| packages/data-provider/src/schemas.ts | Allow tokenCount on messages via zod schema. |
| packages/data-provider/src/keys.ts | Add React Query key for tokenConfig. |
| packages/data-provider/src/data-service.ts | Add getTokenConfig() data-service call. |
| packages/data-provider/src/config.ts | Add interface.contextUsage and interface.contextCost flags/defaults. |
| packages/data-provider/src/api-endpoints.ts | Add /api/endpoints/token-config endpoint helper. |
| packages/api/src/stream/interfaces/IJobStore.ts | Add serialized fields for context snapshot + token-usage replay. |
| packages/api/src/stream/implementations/RedisJobStore.ts | Deserialize persisted contextUsage/tokenUsage. |
| packages/api/src/stream/GenerationJobManager.ts | Persist latest context snapshot + per-call usage for resume. |
| packages/api/src/stream/tests/usageResume.spec.ts | Add unit tests for usage/context resume state. |
| packages/api/src/endpoints/pricing.ts | Implement buildTokenConfigMap() for context/pricing resolution. |
| packages/api/src/endpoints/pricing.spec.ts | Add tests for token-config map resolution behavior. |
| packages/api/src/endpoints/index.ts | Export new pricing/token-config utilities. |
| packages/api/src/endpoints/custom/initialize.ts | Factor out getTokenConfigKey() for user-scoped caching. |
| e2e/specs/mock/usage.spec.ts | Add Playwright mock test for live gauge + reload fallback. |
| e2e/setup/fake-model.js | Emit synthetic provider-style usage_metadata in mock streaming. |
| client/src/utils/tokens.ts | Add module-level token index + branch summation/cost helpers. |
| client/src/utils/tokens.spec.ts | Add unit tests for token index, estimation, and formatting. |
| client/src/utils/messages.ts | Memoize buildTree by message-array identity to avoid repeat builds. |
| client/src/utils/index.ts | Export token utilities from client utils barrel. |
| client/src/store/usage.ts | Add per-conversation Jotai atom families for usage/snapshots/live estimates. |
| client/src/store/index.ts | Export usage store module and reorder imports. |
| client/src/locales/en/translation.json | Add i18n strings for context/cost/usage breakdown UI. |
| client/src/hooks/SSE/useUsageHandler.ts | Add SSE-driven imperative writers for usage/context atoms + token index. |
| client/src/hooks/SSE/useSSE.ts | Wire on_context_usage/on_token_usage events into the SSE flow. |
| client/src/hooks/SSE/useResumableSSE.ts | Restore usage/context/live-estimates from resume state and replay events. |
| client/src/hooks/SSE/index.ts | Export useUsageHandler. |
| client/src/hooks/Chat/useTokenUsage.ts | Build a view-model for gauge rendering using snapshots/estimates. |
| client/src/hooks/Chat/useTokenLimits.ts | Resolve max context + rates via conversation/agent/spec/token-config. |
| client/src/hooks/Chat/index.ts | Export new token usage/limits hooks. |
| client/src/data-provider/Endpoints/queries.ts | Add useTokenConfigQuery() (infinite stale time). |
| client/src/components/Chat/Input/TokenUsage/index.tsx | Add gauge trigger + hover breakdown, gated by interface config. |
| client/src/components/Chat/Input/TokenUsage/Gauge.tsx | Render ring gauge with thresholds + indeterminate mode. |
| client/src/components/Chat/Input/TokenUsage/Breakdown.tsx | Render breakdown popover (context composition, usage, cost). |
| client/src/components/Chat/Input/ChatForm.tsx | Mount TokenUsage next to chat input controls. |
| api/server/routes/endpoints.js | Register /api/endpoints/token-config route. |
| api/server/controllers/TokenConfigController.js | Implement token-config controller with custom endpoint overrides + pricing gate. |
| api/server/controllers/agents/callbacks.js | Emit per-call token usage and forward (guarded) context snapshots. |
| api/server/controllers/agents/tests/usageEvents.integration.spec.js | Add integration coverage for usage/context events + persistence. |
| api/app/clients/BaseClient.js | Stop deleting responseMessage.tokenCount so output token counts propagate. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| const previousWrite = this.replayEventWriteQueues.get(streamId) ?? Promise.resolve(); | ||
| await this.jobStore.updateJob(streamId, { | ||
| contextUsage: JSON.stringify((event as { data?: unknown }).data ?? null), | ||
| }); | ||
| } |
| runCode: z.boolean().optional(), | ||
| webSearch: z.boolean().optional(), | ||
| contextUsage: z.boolean().optional(), | ||
| contextCost: z.boolean().optional(), | ||
| peoplePicker: z |
| /** Jotai atomFamily entries are never GC'd — call on conversation switch/cleanup */ | ||
| export function removeUsageAtoms(conversationId: string): void { | ||
| branchTotalsFamily.remove(conversationId); | ||
| contextSnapshotFamily.remove(conversationId); | ||
| usageTotalsFamily.remove(conversationId); | ||
| liveTokensFamily.remove(conversationId); | ||
| calibrationFamily.remove(conversationId); | ||
| } |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c7f5c38af
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| contextUsage: z.boolean().optional(), | ||
| contextCost: z.boolean().optional(), |
There was a problem hiding this comment.
These new interface options are parsed here, but the resolved server/startup config is built by loadDefaultInterface, which whitelists each interface field and does not include contextUsage or contextCost (packages/data-schemas/src/app/interface.ts). As a result, deployments that set interface.contextUsage: false still render the token indicator, and interface.contextCost: true never reaches TokenConfigController, so pricing is omitted from /api/endpoints/token-config. Add these fields to the loaded interface config so the YAML values survive validation.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in e5ab7cf — added contextUsage/contextCost to loadDefaultInterface so the YAML values survive into the resolved interface config (defaults still flow from the schema).
| await this.emitUsage({ | ||
| input_tokens: taggedUsage.input_tokens, | ||
| output_tokens: taggedUsage.output_tokens, | ||
| total_tokens: taggedUsage.total_tokens, | ||
| input_token_details: taggedUsage.input_token_details, | ||
| model: taggedUsage.model, |
There was a problem hiding this comment.
When a provider reports cache usage in the top-level cache_creation_input_tokens / cache_read_input_tokens fields (the existing spend path already supports this shape for Claude/Bedrock-style metadata), this emitted event drops those fields and only forwards input_token_details. With contextCost enabled the client then calculates zero cache write/read tokens for those calls, and resumed streams persist the same incomplete event. Include or normalize the top-level cache fields before streaming the usage event.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in e5ab7cf — the emitted event now normalizes top-level cache_creation_input_tokens/cache_read_input_tokens into input_token_details, matching the spend path's dual-shape handling.
| const router = express.Router(); | ||
| /** Auth required for role/tenant-scoped endpoint config resolution. */ | ||
| router.get('/', requireJwtAuth, endpointController); | ||
| router.get('/token-config', requireJwtAuth, tokenConfigController); |
There was a problem hiding this comment.
This route only runs requireJwtAuth, and unlike the existing endpoint/model loaders it never populates req.config before TokenConfigController reads it. In that scenario appConfig is undefined, so customEndpoints is always empty and YAML/fetched tokenConfig overrides for custom endpoints are not applied; custom models whose context comes from those overrides return missing context/pricing from /api/endpoints/token-config. Add the config middleware here or have the controller load the app config before resolving endpoint token configs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in e5ab7cf — the token-config route now runs configMiddleware so req.config is populated before the controller resolves custom endpoint overrides.
| const endpoint = submission.conversation?.endpoint ?? ''; | ||
| const model = data.model ?? submission.conversation?.model ?? ''; | ||
| const rates = tokenConfig?.[endpoint]?.[model]; |
There was a problem hiding this comment.
For Agents runs backed by a non-OpenAI provider, submission.conversation.endpoint is agents while the usage event carries the actual provider in data.provider and the token-config rates are keyed under that provider endpoint. Looking up tokenConfig[agents][model] therefore leaves rates undefined for Anthropic/custom agents, so contextCost shows no session cost even though usage events arrive. Prefer the event/provider endpoint for Agents conversations before falling back to the conversation endpoint.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in e5ab7cf — rate lookup now prefers data.provider (the underlying provider for agent runs) before falling back to the conversation endpoint.
| } else if (pendingEvent.event === UsageEvents.ON_TOKEN_USAGE) { | ||
| if (!hasUsageBackfill) { | ||
| usageHandler(pendingEvent.data, resumeSubmission); |
There was a problem hiding this comment.
When resuming an in-memory stream that already has collectedUsage, token-usage events that arrive in the gap after getResumeState() but before the resumed subscription are returned in pendingEvents, not included in the backfill snapshot. This branch skips every pending usage event whenever any backfill exists, so multi-call agent runs can undercount the later model call; only replay events represented by the snapshot should be suppressed, while gap events still need to be folded.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in e5ab7cf — replaced the blanket skip with multiset matching against the backfill snapshot: replayed/gap usage events already in the snapshot are deduped, while events that raced past the snapshot read still fold.
| const customEndpoints = appConfig?.endpoints?.[EModelEndpoint.custom] ?? []; | ||
| const cache = tokenConfigCache(); | ||
| for (const endpointConfig of customEndpoints) { | ||
| const name = endpointConfig?.name; |
There was a problem hiding this comment.
loadConfigModels keys custom model lists by normalizeEndpointName(configName), but this controller stores token-config overrides under the raw YAML name. For configs such as name: Ollama (normalized elsewhere to ollama), buildTokenConfigMap looks up endpointTokenConfigs['ollama'] and misses both YAML and fetched overrides, so those custom models lose their configured context/pricing. Normalize the name before using it as the override/cache key.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in e5ab7cf — the controller now keys overrides by normalizeEndpointName, matching how loadConfigModels and the token-config cache key custom endpoints.
|
@codex review |
|
Codex Review: Didn't find any major issues. What shall we delve into next? ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1843cff325
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const cacheIsAdditive = cacheWrite + cacheRead > input; | ||
| const baseInput = cacheIsAdditive ? input : Math.max(0, input - cacheRead - cacheWrite); |
There was a problem hiding this comment.
When contextCost is enabled, this heuristic underprices providers whose cache tokens are additive but smaller than the non-cached input (for example Anthropic/Bedrock calls with input_tokens: 10000 and cache_read: 4000). The server-side billing path already keys this off usage.provider in packages/api/src/agents/usage.ts (inputTokensIncludesCache) because additive providers should bill the full input_tokens plus cache buckets, while this branch subtracts cache unless cache exceeds input, so the UI can show a materially lower session cost than the actual recorded spend.
Useful? React with 👍 / 👎.
| const tokenKey = getTokenConfigKey(endpointConfig, name, req.user.id); | ||
| const cached = await cache.get(tokenKey); |
There was a problem hiding this comment.
For custom endpoints whose model list is user-scoped (user-provided key/baseURL, or configured headers resolved against the user), the getModelsConfig(req) call above can fetch /models and build token pricing/context data, but fetchModels writes that data under its default name key because loadConfigModels does not pass this user-scoped tokenKey. This lookup then asks for ${endpoint}:${userId} and misses on a cold cache, so /api/endpoints/token-config returns fallback/static data without the fetched context window or pricing for those user-scoped custom endpoints until some other path happens to populate the same key.
Useful? React with 👍 / 👎.
| const snapshotActive = | ||
| snapshot != null && | ||
| (isSubmitting || snapshot.anchorMessageId == null || branchTotals.containsAnchor); | ||
|
|
||
| if (snapshotActive && snapshot) { | ||
| const breakdown = snapshot.breakdown; | ||
| const maxTokens = snapshot.contextBudget ?? breakdown.maxContextTokens; | ||
| const instructionTokens = snapshot.effectiveInstructionTokens ?? breakdown.instructionTokens; | ||
| const baseUsed = | ||
| snapshot.remainingContextTokens != null | ||
| ? maxTokens - snapshot.remainingContextTokens | ||
| : instructionTokens + breakdown.messageTokens; | ||
| const usedTokens = Math.max(0, baseUsed) + liveTokens; |
There was a problem hiding this comment.
After an agent run completes, finalizeUsage upserts the response token count into branchTotals and clears liveTokens, but this branch remains active whenever the just-submitted user message is still on the selected branch. Since the context snapshot was emitted before the LLM call, baseUsed + liveTokens then drops the finalized assistant output from the gauge until a later snapshot or reload, so completed conversations underreport context by the last response's tokens.
Useful? React with 👍 / 👎.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4827ae7b95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const tokenKey = getTokenConfigKey(endpointConfig, name, req.user.id); | ||
| const cached = await cache.get(tokenKey); |
There was a problem hiding this comment.
For custom endpoints with a user-provided key or baseURL, getTokenConfigKey() returns a user-scoped key here, but the preceding getModelsConfig() path fetches those models via createLoadConfigModels without passing tokenKey; fetchModels then stores the derived token config under name (tokenKey ?? name). On a first load of /api/endpoints/token-config, this lookup misses the provider token config that was just fetched and returns only static fallbacks until a chat initializes the endpoint with the scoped key.
Useful? React with 👍 / 👎.
| tokenomics.prompt = overrideRates.prompt; | ||
| tokenomics.completion = overrideRates.completion; |
There was a problem hiding this comment.
When a custom tokenConfig entry includes cache prices, this override branch only exposes prompt and completion. The billing path reads custom cache rates from endpointTokenConfig[model].write/read, while the UI falls back to prompt pricing when cacheWrite/cacheRead are absent, so cached-token costs shown by the new gauge are wrong for custom endpoints that configure different cache read/write rates.
Useful? React with 👍 / 👎.
| const rates = | ||
| (data.provider != null ? tokenConfig?.[data.provider]?.[model] : undefined) ?? | ||
| tokenConfig?.[endpoint]?.[model]; |
There was a problem hiding this comment.
When an Agent uses the Vertex AI provider, usage events carry provider: "vertexai", but the token-config response is populated under the google endpoint key from the default model loader (there is no default vertexai entry). This lookup therefore leaves rates undefined for Vertex/Gemini agent runs (and the same alias affects context-limit lookup), so the new usage/cost UI silently loses pricing and limit data even though it exists under google.
Useful? React with 👍 / 👎.
| if ( | ||
| data.event === StepEvents.ON_MESSAGE_DELTA || | ||
| data.event === StepEvents.ON_REASONING_DELTA | ||
| ) { | ||
| tapStream(data.data, { ...submission, userMessage }); |
There was a problem hiding this comment.
This only feeds tapStream from agent step delta events. Standard chat streams handled by the later data.type/text branches (OpenAI, custom OpenAI-compatible endpoints, etc.) never call tapStream, so the newly mounted token gauge does not update during the response for those common endpoints and only changes after the final message is saved.
Useful? React with 👍 / 👎.
GitNexus: 🚀 deployedThe |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05e48ec583
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const cacheIsAdditive = cacheWrite + cacheRead > input; | ||
| const baseInput = cacheIsAdditive ? input : Math.max(0, input - cacheRead - cacheWrite); |
There was a problem hiding this comment.
When contextCost is enabled, this size-based heuristic under-reports session cost for additive-cache providers such as Anthropic/Bedrock whenever their cache tokens are smaller than the non-cached input (for example input_tokens=1000, cache_read=100 is charged as only 900 prompt tokens here). The backend billing path does not infer this from relative sizes; packages/api/src/agents/usage.ts treats only known subset providers as including cache in input_tokens, and otherwise charges input + write + read, so the UI can show a materially lower cost than the actual transactions for common cached Claude calls.
Useful? React with 👍 / 👎.
| const tokenKey = getTokenConfigKey(endpointConfig, name, req.user.id); | ||
| const cached = await cache.get(tokenKey); |
There was a problem hiding this comment.
For custom endpoints whose API key/base URL or forwarded headers are user-scoped, getTokenConfigKey() returns ${endpoint}:${userId} here, but the getModelsConfig(req) call that just fetched those models invokes fetchModels without passing that token key (see packages/api/src/endpoints/config/models.ts), so fetchModels stores the generated token config under the plain endpoint name. As a result /api/endpoints/token-config omits fetched context/pricing for those endpoints until some other chat initialization happens to populate the user-scoped key, leaving the new gauge without the server-resolved limits for user-key/header based custom providers.
Useful? React with 👍 / 👎.
|
|
||
| const foldUsage = (data: TTokenUsageEvent, submission: UsageSubmissionLike) => { | ||
| const convoKey = getConvoKey(submission); | ||
| const tokenConfig = queryClient.getQueryData<TTokenConfigMap>([QueryKeys.tokenConfig]); |
There was a problem hiding this comment.
When contextCost is enabled, this synchronous cache read can be undefined for the first fast on_token_usage event because the /token-config query is started by the gauge at the same time as the stream. That event is then folded into usageTotals with calcUsageCost(..., undefined) === 0 and is never recomputed when the token-config query finishes, so short/first model calls can permanently show $0.00 or undercount the session cost even though pricing is enabled.
Useful? React with 👍 / 👎.
| } | ||
| } | ||
| } | ||
| registry.set(conversationId, index); |
There was a problem hiding this comment.
This module-level registry keeps a full message-token map for every conversation passed to buildIndex, but production code never calls the exported clearIndex cleanup path (repo-wide rg "clearIndex\(" only finds the export and tests). In a long-lived tab that opens many conversations, these maps retain all visited message IDs/token entries even after React Query and the Jotai atoms are removed, causing avoidable memory growth proportional to browsing history.
Useful? React with 👍 / 👎.
| return 0; | ||
| } | ||
| const input = usage.input_tokens ?? 0; | ||
| const output = usage.output_tokens ?? 0; |
There was a problem hiding this comment.
For providers where usage_metadata.output_tokens omits billable reasoning/thinking tokens but total_tokens includes them, this charges only the raw output field. The backend billing path repairs that case in resolveCompletionTokens by using total_tokens - input_tokens (after cache adjustment), so Vertex/Gemini-style calls with hidden reasoning can show a lower session cost than the actual transaction total.
Useful? React with 👍 / 👎.
| tokenomics.prompt = deps.getMultiplier({ | ||
| valueKey, | ||
| model, | ||
| endpoint, | ||
| tokenType: 'prompt', | ||
| }); |
There was a problem hiding this comment.
These server-resolved rates are computed once without an inputTokenCount, so models with premium thresholds always expose their base prompt/completion multipliers to the client. For a long-context call that crosses a tiered threshold (for example the premium entries handled by getMultiplier when billing transactions), the UI will still multiply by the cheaper base rate and understate the session cost even though contextCost is intended to mirror backend accounting.
Useful? React with 👍 / 👎.
GitNexus: ❌ deploy failedThe deploy failed — the previous index (if any) continues to be served. |
GitNexus: ❌ deploy failedThe deploy failed — the previous index (if any) continues to be served. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7489c2be8c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const tokenKey = getTokenConfigKey(endpointConfig, name, req.user.id); | ||
| const cached = await cache.get(tokenKey); |
There was a problem hiding this comment.
For custom endpoints with models.fetch plus a user-provided API key/base URL or user-scoped forwarded headers, getTokenConfigKey() returns a user-scoped key here, but the preceding getModelsConfig() fetch path stores the fetched token config under the endpoint name because it does not pass this tokenKey into fetchModels. In those deployments this lookup misses the token config that was just fetched, so /api/endpoints/token-config falls back to static/default values for custom models and the UI shows missing or wrong context/pricing until some other chat initialization happens to populate the scoped key.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 84436aa — the controller now falls back from the user-scoped key to the plain endpoint name, accepting whichever writer populated the cache (models-config fetch stores plain; chat initialization stores scoped).
| handle: async (event, data) => { | ||
| await emitEvent(res, streamId, { event, data }); |
There was a problem hiding this comment.
When a sequential agent run sets hide_sequential_outputs, the message/reasoning delta handlers gate emissions with checkIfLastAgent(...) || !metadata?.hide_sequential_outputs, but this new context-usage handler forwards every ON_CONTEXT_USAGE event unconditionally. Hidden intermediate agents can therefore still update and persist the client's context gauge with their token budget snapshots, exposing or confusing information that the stream otherwise suppresses; apply the same visibility predicate before emitting.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 84436aa — the context-usage forwarder now applies the same visibility predicate as the delta handlers (checkIfLastAgent || !hide_sequential_outputs), so hidden intermediate agents no longer update the client gauge.
| } | ||
| /> | ||
| <div className="mx-auto flex" /> | ||
| <TokenUsage index={index} conversation={conversation} isSubmitting={isSubmitting} /> |
There was a problem hiding this comment.
When a user changes the active conversation's max context setting, this new gauge can stay on the old limit because ChatFormWrapper passes a memoized stableConversation whose dependency list does not include conversation.maxContextTokens, while useTokenLimits() now reads that field to compute maxContextTokens. Until some unrelated dependency such as model or endpoint changes, the gauge keeps rendering with the previous budget.
Useful? React with 👍 / 👎.
| data.event === StepEvents.ON_MESSAGE_DELTA || | ||
| data.event === StepEvents.ON_REASONING_DELTA | ||
| ) { | ||
| tapStream(data.data, { ...currentSubmission, userMessage }); |
There was a problem hiding this comment.
After tapStream starts accumulating live output, a dropped connection that reconnects after the backend has already completed and deleted the job takes the existing 404 path, which invalidates/refetches messages but never calls resetLive or finalizeUsage. In that case the refetched response's persisted tokenCount is added to the stale live estimate, so the context gauge over-counts until another stream resets it.
Useful? React with 👍 / 👎.
|
@codex review |
|
@codex review |
- raise the popup breakdown's context progressbar contrast (muted surface-tertiary track, prominent text-primary fill) — that's the bar the contrast feedback was about - revert the gauge ring restyle (kept its original border-heavy track / text-secondary fill); the ring wasn't the element in question
… Memo - a null-anchor context snapshot was treated as active on every branch, leaking one generation's granular breakdown onto sibling branches. Require a non-null (response-message) anchor on the viewed branch instead, so siblings without a matching snapshot fall back to their own totals. - revert the buildTree WeakMap memo in messages.ts. buildTree is pure (builds from shallow copies) so the memo was behaviorally identical, but it was the feature's only change to core branch-navigation selectors — removing it matches upstream and rules it out of branch-navigation debugging.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0de3f927d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| endpointTokenConfig = toBillingTokenConfig( | ||
| endpointConfig.tokenConfig as Record<string, Record<string, number>>, | ||
| ); |
There was a problem hiding this comment.
When a custom endpoint defines a static tokenConfig for only some of its models, this partial map becomes the endpointTokenConfig used by runtime model initialization. getModelMaxTokens(..., endpointTokenConfig) treats any provided map as authoritative and does not fall back to the built-in maxTokensMap, so an unlisted built-in model on the same endpoint can drop to the default context budget even though /token-config advertises the built-in fallback. This over-prunes or mis-budgets those unlisted models; keep the same fallback behavior in runtime max-token resolution or only apply the override when the selected model matches it.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 859bf37. `getModelMaxTokens` / `getModelMaxOutputTokens` now try the override first and fall back to the built-in `maxTokensMap` / `maxOutputTokensMap` for any model a partial override does not list (`system_default` precedence preserved). This matches what `/token-config` advertises via `buildTokenConfigMap`, so unlisted built-in models on the same endpoint keep their real context budget instead of dropping to the default. Also consolidated the builder's now-redundant manual fallback into the single call, and added `tokens.spec.ts` coverage for the partial-override case.
| if (usageCost?.enabled === true && usageCost.pricing) { | ||
| try { | ||
| payload = { ...data, cost: computeUsageCostUSD(data, usageCost.pricing) }; | ||
| } catch (err) { |
There was a problem hiding this comment.
When interface.contextCost is enabled for a custom/OpenRouter model whose rates come from tokenConfig, this computes the streamed cost with only the global pricing functions. The token-config response uses the endpoint-specific rates, so on_token_usage can fall back to default/static pricing for those models and show an incorrect session cost in the UI. Thread the resolved endpoint token config through usageCost and pass it to computeUsageCostUSD.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 859bf37. `usageCost` now carries `endpointTokenConfig` (set from `primaryConfig.endpointTokenConfig` once the agent resolves) and `computeUsageCostUSD(data, usageCost.pricing, usageCost.endpointTokenConfig)` prices with the endpoint-specific rates, so the streamed `on_token_usage` cost matches billing for custom/OpenRouter models with static `tokenConfig`.
| endpointTokenConfig = toBillingTokenConfig( | ||
| endpointConfig.tokenConfig as Record<string, Record<string, number>>, | ||
| ); |
There was a problem hiding this comment.
For Agents backed by a custom endpoint with the newly supported static tokenConfig, this branch creates the billing override, but initializeAgent only consumes options.endpointTokenConfig for max-context resolution and the later AgentClient is never constructed with it; AgentClient.recordCollectedUsage therefore sees this.options.endpointTokenConfig as undefined. In that scenario transactions and balance deductions use the global/default rates instead of the admin-configured rates, so pass the resolved config through to the AgentClient spending path as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 859bf37. `initializeAgent` now surfaces `options.endpointTokenConfig` onto the returned `InitializedAgent`, and `AgentClient` is constructed with `endpointTokenConfig: primaryConfig.endpointTokenConfig`, so `recordCollectedUsage` (and the subagent usage emitter, which reads `this.options.endpointTokenConfig`) spend at the admin-configured rates instead of defaults. This supersedes my earlier decline on the same threading — the config genuinely was not reaching the spending path.
…xt Limits Custom-endpoint agents resolve an endpointTokenConfig during agent init but it never reached the AgentClient, so spending, emitted cost, and runtime max-token resolution all fell back to default rates for those agents. - Surface options.endpointTokenConfig on the returned InitializedAgent. - Pass it to the AgentClient (this.options.endpointTokenConfig) so the spending path bills at configured rates. - Thread it through usageCost to computeUsageCostUSD so emitted per-event cost matches billing. - getModelMaxTokens/getModelMaxOutputTokens fall back to the built-in map for models absent from a partial override (matches buildTokenConfigMap); consolidates the duplicated fallback in pricing.ts.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 859bf37eb7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| seedLive( | ||
| countTrailingOutputChars(data.resumeState?.aggregatedContent), | ||
| resumeSubmission, | ||
| ); |
There was a problem hiding this comment.
When resuming during a tool loop after the next on_context_usage snapshot has already been emitted, that snapshot's messageTokens already includes the prior model output/tool turn. This unconditional seedLive(countTrailingOutputChars(aggregatedContent)) then counts the trailing text before the tool part again, so reconnecting in that window shows inflated context usage until another snapshot/finalization resets it. Only seed trailing output when there is no resumed context snapshot, or track output produced after that snapshot.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 6a8e7d7. The trailing-output seed now only runs when the resume carries no context snapshot. When a snapshot is present its `messageTokens` already accounts for produced output (and `contextHandler` resets the live estimate to 0), so the unconditional `seedLive` was double-counting the trailing text until the next snapshot/finalize. Any output streamed after the snapshot but before reconnect is picked back up by `tapStream` as deltas resume, and corrected at finalize.
| : undefined, | ||
| }; | ||
| if (streamId) { | ||
| GenerationJobManager.emitChunk(streamId, { event: UsageEvents.ON_TOKEN_USAGE, data }); |
There was a problem hiding this comment.
For resumable streams with subagent calls, GenerationJobManager.emitChunk is async because it persists the usage event and publishes it; firing it without awaiting means generation can reach final cleanup and delete the job before this subagent usage is stored, and any Redis/publish failure will also bypass the surrounding try/catch. This makes resumed clients miss subagent usage/cost even though the same usage is billed server-side.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 6a8e7d7. The subagent emitter is now `async` and `await`s `GenerationJobManager.emitChunk` inside its `try/catch`, matching every other emitChunk caller (ActionService/MCP) — emitChunk commits the HSET before publishing, so awaiting keeps a Redis/publish failure inside the catch instead of surfacing as an unhandled rejection, and the persist completes within the emitter rather than floating into job cleanup. Note billing is unaffected regardless: the child usage is pushed to `collectedUsage` synchronously in the sink (before `onUsage`), so `recordCollectedUsage` bills it and the resume path recovers it via `backfillUsage(resumeState.collectedUsage)` independent of the emit.
| subagentAggregatorsByToolCallId, | ||
| /** Resolved endpoint token/pricing config so spending and cost reflect | ||
| * configured rates for custom-endpoint agents instead of defaults. */ | ||
| endpointTokenConfig: primaryConfig.endpointTokenConfig, |
There was a problem hiding this comment.
When a graph contains connected/added agents that use a different endpoint or provider, passing only the primary agent's endpointTokenConfig into AgentClient makes both recordCollectedUsage and the emitted cost calculation price every collected usage item with the primary custom endpoint's overrides. For example, a secondary agent using the same model id on another endpoint can be over/under-charged by the primary endpoint's static tokenConfig; the per-agent resolved config needs to travel with each usage item or be selected by provider/agent.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Valid for the narrow case; deferring the full fix. Context: before this PR the agents path never passed `endpointTokenConfig` to `AgentClient` at all, so all usage (primary included) was billed at default rates. Threading `primaryConfig.endpointTokenConfig` (859bf37) fixes the common single-custom-endpoint-agent case and does not regress distinct-model-id secondaries — a model absent from the primary's `tokenConfig` falls back to the built-in rate map exactly as before. The only affected scenario is a secondary agent on a different endpoint that shares the same model id as a primary `tokenConfig` entry with different rates. Fully per-agent pricing needs each collected usage item to carry its agent's resolved config and a per-item resolver threaded through the shared `recordCollectedUsage` (also used by responses.js/openai.js) — tracked as a focused follow-up rather than expanding this PR's billing surface.
The granular context breakdown lives only in the live on_context_usage snapshot — a single per-conversation slot, anchored to the latest response and overwritten by each generation. Switching to a branch generated earlier this session lost its tool/skill/system rows and fell back to coarse totals. Retain each generation's finalized snapshot in a per-conversation map keyed by its branch-unique response id (snapshotsByAnchorFamily). When the live snapshot is off the viewed branch, walk the branch tail for its deepest stored anchor and render that breakdown. Bounded by generation count and cleared on conversation switch; the live/just-generated path is unchanged.
- useResumableSSE: skip the trailing-output live seed when the resume carries a context snapshot; the snapshot's messageTokens already counts produced output, so seeding it again inflated usage until the next reset. - AgentClient subagent emitter: await GenerationJobManager.emitChunk like every other caller (it persists before publishing), so a floating promise can't race job cleanup and a Redis/publish failure is caught by the emitter's try/catch instead of surfacing as an unhandled rejection.
|
@codex review |
1 similar comment
|
@codex review |
GitNexus: ❌ deploy failedThe deploy failed — the previous index (if any) continues to be served. |
Add a test-only data-testid distinguishing the granular snapshot breakdown (context-breakdown) from the coarse message-history estimate (context-estimate), then assert granularity in the mock e2e harness: - renders the granular breakdown from the live on_context_usage snapshot (guards that the snapshot event actually reaches the popover, not just the usage totals). - preserves the granular breakdown after switching branches — regenerate to overwrite the single live snapshot, switch back, and confirm the rows survive via the per-anchor snapshot history map. Branch regenerate/sibling selectors mirror the existing chat.spec branch test. All three usage specs pass against the mock pipeline.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f2c302e4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (data.resumeState?.contextUsage) { | ||
| contextHandler(data.resumeState.contextUsage, resumeSubmission); | ||
| } else { |
There was a problem hiding this comment.
When resuming an active stream after on_context_usage has been persisted, this branch skips seedLive(...). The stored context snapshot is the pre-invoke budget, while text already streamed during the disconnected window is only restored through aggregatedContent; backfilled usage updates totals but not confirmedRef, and Redis/cross-replica resumes do not replay those deltas as pending events. As a result, the context gauge undercounts already-produced output during the resumed run (and can finalize the snapshot with too little completedOutputTokens when the usage event was only backfilled).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1ac6027 — and you are right that my previous gate over-corrected. Root cause: `countTrailingOutputChars` skipped a trailing tool-call part and counted the text before it, which is already folded into the next snapshot's `messageTokens`. It now counts only the contiguous output at the very END of the aggregated content (0 when the latest part is a tool call), and the resume path always seeds it again. So in-flight output the pre-invoke snapshot does not yet cover is restored, while a snapshot's budget is never double-counted. The only residual is a tiny window (resumed after a call emitted a tool_call but before the next snapshot) — bounded to one call's text and corrected at the next snapshot/finalize.
| /** Resolved endpoint token/pricing config so spending and cost reflect | ||
| * configured rates for custom-endpoint agents instead of defaults. */ | ||
| endpointTokenConfig: primaryConfig.endpointTokenConfig, |
There was a problem hiding this comment.
In a multi-agent run where a connected or added agent uses a different custom endpoint/tokenConfig than the primary agent, only primaryConfig.endpointTokenConfig is passed into AgentClient. The collected usage events for secondary agents still flow through recordCollectedUsage and the subagent cost emitter with this single config, so those calls are billed/displayed with the primary endpoint's rates or default rates instead of the secondary agent's configured rates. Thread the resolved config per usage-producing agent rather than using only the primary config.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Holding this as a deferred follow-up (chip filed), consistent with the prior round. To be clear on scope: before this PR the agents path never passed `endpointTokenConfig` to `AgentClient` at all, so all usage was billed at default rates. Threading `primaryConfig.endpointTokenConfig` fixes the common single-custom-endpoint-agent case with no regression for distinct-model-id secondaries (a model absent from the primary config falls back to the built-in rate map exactly as before). The only affected case is a secondary agent on a different endpoint sharing the same model id as a primary `tokenConfig` entry. Fully per-agent pricing needs each collected usage item to carry its agent's resolved config and a per-item resolver threaded through the shared `recordCollectedUsage` (also used by responses.js/openai.js) — a billing-pipeline change I'd rather not fold into this PR.
| if (isSubmittingRef.current || event.type !== 'updated') { | ||
| return; |
There was a problem hiding this comment.
When the backend does not emit context snapshots (for example the documented pre-@librechat/agents@3.2.35 fallback path, or non-agent streams), skipping all message-query updates while isSubmitting leaves the module token index on the pre-submit branch. After the created event changes the active tail to the placeholder response, sumBranch cannot find that tail and the gauge shows only live output tokens, dropping the existing history and current prompt until the final event calls finalizeUsage.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1ac6027. `useTokenUsage` now re-indexes from the messages cache on tail change while `isSubmitting`. The cache subscriber is intentionally muted during streaming (no per-chunk O(n) rebuilds), but the `created` event still moves the tail to the placeholder response — so in the no-snapshot fallback (non-agent streams / pre-3.2.35) `sumBranch` missed that tail and dropped history + prompt until `finalizeUsage`. The re-index is bounded: `tailId` only shifts on created/finalize/branch-switch, never per chunk, so streaming stays O(1) per delta.
| /** Lets the host stream the billed child usage to the client (tagged | ||
| * `subagent`, so it folds into session cost/totals but not the live | ||
| * gauge) — child runs never reach ModelEndHandler's emit path. */ | ||
| onUsage?.(usage); |
There was a problem hiding this comment.
For resumable streams, buildSubagentUsageEmitter is async because it awaits GenerationJobManager.emitChunk, but the sink fires it without awaiting or retaining the promise. In subagent-heavy runs this lets the graph proceed to the final event and job cleanup while the on_token_usage emit/persist is still pending, so the UI/resume state can miss billed subagent usage even though collectedUsage was charged. Make the sink/host await these emissions or flush them before finalizing.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1ac6027. `AgentClient` now retains each subagent emit promise (`pendingSubagentEmits`) and flushes them with `Promise.allSettled` in `chatCompletion`'s finally, before the response returns and the job is cleaned up. You're right that the emitter being async was not enough — the sink fires it without awaiting, and resume reads the usage that `emitChunk` persists (HSET before publish via `trackTokenUsage`), so cleanup racing the persist would drop billed subagent usage from resumed clients. (Live billing was always safe — `collectedUsage` is pushed synchronously in the sink.)
GitNexus: ❌ deploy failedThe deploy failed — the previous index (if any) continues to be served. |
… Flush Codex round on the prior commit: - countTrailingOutputChars now counts only output at the very END of the aggregated content (0 when the model paused at a tool call), and the resume path always seeds it. The earlier skip-trailing-tool-parts behavior plus the skip-seed-when-snapshot gate together over- or under-counted in-flight output on resume; one rule fixes both — pre-invoke snapshot budget is never double-counted, and genuine in-flight output is no longer dropped. - useTokenUsage re-indexes from the messages cache on tail change while submitting. The cache subscriber is muted during streaming, so without a context snapshot (non-agent streams) sumBranch missed the created tail and dropped history + prompt until finalize. Bounded — tailId only shifts on created/finalize/branch-switch. - AgentClient tracks subagent usage emit promises and flushes them in chatCompletion's finally. The sink fires the emitter without awaiting, and resume reads the usage emitChunk persists (HSET), so cleanup must not race it or resumed clients miss billed subagent usage.
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
GitNexus: ❌ deploy failedThe deploy failed — the previous index (if any) continues to be served. |
…13670) * 📊 feat: Real-Time Context Window & Token Usage Tracking * 🧪 fix: Align Pricing Spec Dep Signatures with TxDeps * 🩹 fix: Resolve Codex Findings for Context Usage Tracking * 📊 feat: Granular Tool Token Breakdown with Deferred Splits * 🧪 test: Cover Session Cost in Mock E2E and Scope Usage Selectors * 🧪 test: Live Host-Pipeline Usage Verification (Env-Gated) * 🧪 test: Local Real-Provider Multi-Turn E2E Harness * 🪙 fix: Keep Tagged Usage Buckets Out of the Live Context Estimate * 🩹 fix: Scoped Token-Config Fallback and Sequential Visibility for Usage Events * 🩹 fix: Address Usage Review Findings — Cost Timing, Scoped Caches, Finalized Output - carry the post-snapshot output estimate into the context snapshot at finalize so the gauge keeps the last response after live resets - accumulate per-rate billable units and price the session cost at render, so usage events arriving before the token-config load still count once it resolves - pass user-scoped token-config cache keys through loadConfigModels fetches and drop the controller's unscoped fallback to prevent serving another user's resolved config - tag emitted usage events with a per-run seq so resume dedupe never drops a distinct call with an identical payload - admit the static tokenConfig override in the custom endpoint schema so it survives zod parsing into req.config * 🩹 fix: Align Client Usage Accounting with Backend Cost Semantics - classify cache tokens by provider (shared inputTokensIncludesCache from data-provider, consumed by both the backend billing path and the client) instead of a magnitude heuristic, so Anthropic/Bedrock turns where cache is smaller than uncached input no longer under-bill input - mirror resolveCompletionTokens on the client so Vertex-style hidden thinking tokens are reflected in the Output row and session cost - prefer endpoint pricing over adapter-provider pricing so a custom endpoint can price a known model name without built-in rates shadowing it - carry static cacheRead/cacheWrite overrides through the tokenConfig schema and buildTokenConfigMap * 🩹 fix: Honor Static Token Config in Billing; Tighten Usage Freshness - initializeCustom now uses a static endpoint tokenConfig as the agent's endpointTokenConfig (billing + balance checks), not just the advertised UI config — previously the gauge showed admin rates while the agent billed against built-in tables - invalidate the token-config query alongside models on user-key add/ revoke so context windows and pricing refresh without a reload - include maxContextTokens in ChatForm's stabilized conversation memo so the gauge reflects a changed context-window setting immediately - feed the live output estimate from the legacy content path (direct and assistants streams), setting from cumulative part text rather than accumulating deltas * 🩹 fix: Resume Usage Dedup, Agent Pricing, and Partial Override Billing - fold usage events idempotently by (runId, seq) so resume backfill no longer resets the conversation totals — a mid-stream reconnect keeps the usage of prompts already completed earlier in the session - tap replayed pending message/reasoning/content events so output streamed past the resume snapshot reaches the live estimate, not just the message - resolve cost against the agent's backing endpoint (Agents conversations report endpoint `agents` / provider `openAI`, neither of which keys a custom endpoint's tokenConfig) - getMultiplier/getCacheMultiplier fall back to the standard tables for models absent from a partial endpointTokenConfig, so a partial static override no longer bills non-listed models at defaultRate while the UI shows the correct pattern rate * 🩹 fix: Repaired Output in Gauge, Cache-Rate Keys, Config Gate, Usage Cleanup - live/completed gauge counts the repaired completion (normalized output), so under-reporting providers don't drop the response from used context - translate static tokenConfig cacheWrite/cacheRead onto the write/read keys getCacheMultiplier reads, so cache tokens bill at the configured rate instead of the prompt-rate fallback - clear the token index and usage atoms when leaving a conversation, so visited histories don't accumulate in memory for the tab's lifetime - wait for startupConfig before mounting the gauge, so a deployment with contextUsage disabled never briefly mounts it or fires the token-config query on first load * 🩹 fix: Move Token-Config Resolution to TS; Key Live Usage by Created Convo - extract the token-config resolution (override gathering + cache lookup + buildTokenConfigMap) into resolveTokenConfigMap in packages/api, leaving the /api controller a thin request-scoped wrapper (CLAUDE.md TS rule) - getConvoKey prefers the user message's real conversationId once the `created` event stamps it, so a new chat's first-response live gauge and totals land under the id TokenUsage subscribes to instead of NEW_CONVO * 🩹 fix: Clear Stale Redis Job Usage; Live-Tap Legacy Streams; Share Fetched Config - DEL the Redis job hash before re-creating it so a reused streamId can't inherit a prior run's contextUsage/tokenUsage and backfill stale usage - tap the legacy {message,text} stream branch (non-agent OpenAI/Anthropic streams) into the live estimate, not just the content path - copy a deduped fetch's token config to every sibling endpoint sharing the baseURL/key/headers, so /token-config resolves each by its own name * ⏪ revert: Don't DEL Redis job hash in createJob (breaks cross-replica resume) createJob is an idempotent join — a second replica calls it for the same streamId to share an in-flight stream's state. DELeting the hash wiped the prior replica's persisted created/usage state, so a joining replica missed the created event (GenerationJobManager cross-replica integration test). Reverts the F1 change from 2bfce0c; the stale-usage concern doesn't arise in practice (streamId is unique per generation). * 🩹 fix: Best-Effort Usage Emit; Tag Hidden Sequential-Agent Usage - wrap the ModelEndHandler usage emit in try/catch so a failed telemetry delivery (closed SSE / Redis publish error) can't abort the handler before thought-signature capture, which would break resumed tool calls - tag hidden sequential-agent usage as 'sequential' (non-primary) so the client folds it into session cost/totals but not the live context gauge, instead of letting an undefined usage_type inflate the visible gauge * 🩹 fix: Refetch Stale Token Config on Mount; Normalize Vertex for Lookup - useTokenConfigQuery refetches on mount when stale, so a user-key change that invalidates tokenConfig while the gauge is unmounted takes effect on return instead of serving the prior key's resolved config - normalize a Vertex-backed agent's provider (vertexai) to the google token-config key, so Gemini context windows and rates resolve instead of showing unknown context / $0 cost * ✨ feat: Server-Side Per-Event Cost (Authoritative Pricing for the Gauge) Move usage-cost pricing to the single source of truth. The backend prices each model call with the same billing functions (premium tiers via getMultiplier(inputTokenCount), cache rates) and emits the USD cost on on_token_usage when interface.contextCost is enabled; the client sums emitted costs instead of re-deriving from base token-config rates. - computeUsageCostUSD reuses prepareTokenSpend/prepareStructuredTokenSpend so the emitted cost matches what is billed (incl. premium thresholds) - getDefaultHandlers gains a usageCost pricing context; initialize.js wires db.getMultiplier/getCacheMultiplier gated on contextCost (agents path) - client UsageTotals carries a summed costUSD; retire the client-side rate lookups (costFromUnits/calcUsageCost) that drifted from backend pricing and produced the provider-keying / cache-key / Vertex / premium findings - keep normalizeUsageUnits for the displayed token counts; token-config is still used for the context-window meter Fixes the premium-tier session-cost under-report (gpt-5.x / gemini-3.1 above their input thresholds). * 🩹 fix: Branch-Accurate Usage Snapshot + Clearer Gauge Track Contrast - re-anchor the context snapshot from the user message to the response message at finalize. Regenerating a response branches off a shared user message, so anchoring on it made the snapshot read as "active" on both branches — switching to the sibling branch showed the wrong (other branch's) context. The response message is branch-unique, so sibling branches now correctly fall back to their own per-branch totals. - raise the gauge ring's track/fill contrast (muted track, prominent fill) so the used portion reads clearly as a fill-level indicator * 🩹 fix: Tag Sequential Usage in Billing; Emit Subagent Cost; Reset Live on Resume Errors - tag hidden sequential-agent usage `usage_type: 'sequential'` on the COLLECTED usage (not just the emit), and treat it as non-primary in recordCollectedUsage (billed, excluded from the reported output total) so hidden intermediate output stops inflating the parent's tokenCount/pruning - emit on_token_usage from the subagent usage sink (tagged `subagent`, with authoritative cost when contextCost is on) so the gauge's session cost/totals include billed subagent usage; it stays out of the live meter - call resetLive on the resumable 404 and max-retry terminal branches so the gauge doesn't keep counting stale in-flight tokens after the stream ends * 🎨 fix: Contrast the Popup Context Bar; Revert Ring Restyle - raise the popup breakdown's context progressbar contrast (muted surface-tertiary track, prominent text-primary fill) — that's the bar the contrast feedback was about - revert the gauge ring restyle (kept its original border-heavy track / text-secondary fill); the ring wasn't the element in question * 🩹 fix: Stop Snapshot Granularity Leaking Across Branches; Revert Tree Memo - a null-anchor context snapshot was treated as active on every branch, leaking one generation's granular breakdown onto sibling branches. Require a non-null (response-message) anchor on the viewed branch instead, so siblings without a matching snapshot fall back to their own totals. - revert the buildTree WeakMap memo in messages.ts. buildTree is pure (builds from shallow copies) so the memo was behaviorally identical, but it was the feature's only change to core branch-navigation selectors — removing it matches upstream and rules it out of branch-navigation debugging. * 🪙 fix: Thread Endpoint Token Config to Agent Billing, Cost, and Context Limits Custom-endpoint agents resolve an endpointTokenConfig during agent init but it never reached the AgentClient, so spending, emitted cost, and runtime max-token resolution all fell back to default rates for those agents. - Surface options.endpointTokenConfig on the returned InitializedAgent. - Pass it to the AgentClient (this.options.endpointTokenConfig) so the spending path bills at configured rates. - Thread it through usageCost to computeUsageCostUSD so emitted per-event cost matches billing. - getModelMaxTokens/getModelMaxOutputTokens fall back to the built-in map for models absent from a partial override (matches buildTokenConfigMap); consolidates the duplicated fallback in pricing.ts. * 🪙 fix: Preserve Granular Breakdown Across Branch Switches The granular context breakdown lives only in the live on_context_usage snapshot — a single per-conversation slot, anchored to the latest response and overwritten by each generation. Switching to a branch generated earlier this session lost its tool/skill/system rows and fell back to coarse totals. Retain each generation's finalized snapshot in a per-conversation map keyed by its branch-unique response id (snapshotsByAnchorFamily). When the live snapshot is off the viewed branch, walk the branch tail for its deepest stored anchor and render that breakdown. Bounded by generation count and cleared on conversation switch; the live/just-generated path is unchanged. * 🪙 fix: Harden Resume Seeding and Subagent Usage Emission - useResumableSSE: skip the trailing-output live seed when the resume carries a context snapshot; the snapshot's messageTokens already counts produced output, so seeding it again inflated usage until the next reset. - AgentClient subagent emitter: await GenerationJobManager.emitChunk like every other caller (it persists before publishing), so a floating promise can't race job cleanup and a Redis/publish failure is caught by the emitter's try/catch instead of surfacing as an unhandled rejection. * 🧪 test: Playwright Coverage for Context Breakdown Granularity Add a test-only data-testid distinguishing the granular snapshot breakdown (context-breakdown) from the coarse message-history estimate (context-estimate), then assert granularity in the mock e2e harness: - renders the granular breakdown from the live on_context_usage snapshot (guards that the snapshot event actually reaches the popover, not just the usage totals). - preserves the granular breakdown after switching branches — regenerate to overwrite the single live snapshot, switch back, and confirm the rows survive via the per-anchor snapshot history map. Branch regenerate/sibling selectors mirror the existing chat.spec branch test. All three usage specs pass against the mock pipeline. * 🪙 fix: Correct Resume Live-Seed, Fallback Re-index, and Subagent Emit Flush Codex round on the prior commit: - countTrailingOutputChars now counts only output at the very END of the aggregated content (0 when the model paused at a tool call), and the resume path always seeds it. The earlier skip-trailing-tool-parts behavior plus the skip-seed-when-snapshot gate together over- or under-counted in-flight output on resume; one rule fixes both — pre-invoke snapshot budget is never double-counted, and genuine in-flight output is no longer dropped. - useTokenUsage re-indexes from the messages cache on tail change while submitting. The cache subscriber is muted during streaming, so without a context snapshot (non-agent streams) sumBranch missed the created tail and dropped history + prompt until finalize. Bounded — tailId only shifts on created/finalize/branch-switch. - AgentClient tracks subagent usage emit promises and flushes them in chatCompletion's finally. The sink fires the emitter without awaiting, and resume reads the usage emitChunk persists (HSET), so cleanup must not race it or resumed clients miss billed subagent usage.
…13670) * 📊 feat: Real-Time Context Window & Token Usage Tracking * 🧪 fix: Align Pricing Spec Dep Signatures with TxDeps * 🩹 fix: Resolve Codex Findings for Context Usage Tracking * 📊 feat: Granular Tool Token Breakdown with Deferred Splits * 🧪 test: Cover Session Cost in Mock E2E and Scope Usage Selectors * 🧪 test: Live Host-Pipeline Usage Verification (Env-Gated) * 🧪 test: Local Real-Provider Multi-Turn E2E Harness * 🪙 fix: Keep Tagged Usage Buckets Out of the Live Context Estimate * 🩹 fix: Scoped Token-Config Fallback and Sequential Visibility for Usage Events * 🩹 fix: Address Usage Review Findings — Cost Timing, Scoped Caches, Finalized Output - carry the post-snapshot output estimate into the context snapshot at finalize so the gauge keeps the last response after live resets - accumulate per-rate billable units and price the session cost at render, so usage events arriving before the token-config load still count once it resolves - pass user-scoped token-config cache keys through loadConfigModels fetches and drop the controller's unscoped fallback to prevent serving another user's resolved config - tag emitted usage events with a per-run seq so resume dedupe never drops a distinct call with an identical payload - admit the static tokenConfig override in the custom endpoint schema so it survives zod parsing into req.config * 🩹 fix: Align Client Usage Accounting with Backend Cost Semantics - classify cache tokens by provider (shared inputTokensIncludesCache from data-provider, consumed by both the backend billing path and the client) instead of a magnitude heuristic, so Anthropic/Bedrock turns where cache is smaller than uncached input no longer under-bill input - mirror resolveCompletionTokens on the client so Vertex-style hidden thinking tokens are reflected in the Output row and session cost - prefer endpoint pricing over adapter-provider pricing so a custom endpoint can price a known model name without built-in rates shadowing it - carry static cacheRead/cacheWrite overrides through the tokenConfig schema and buildTokenConfigMap * 🩹 fix: Honor Static Token Config in Billing; Tighten Usage Freshness - initializeCustom now uses a static endpoint tokenConfig as the agent's endpointTokenConfig (billing + balance checks), not just the advertised UI config — previously the gauge showed admin rates while the agent billed against built-in tables - invalidate the token-config query alongside models on user-key add/ revoke so context windows and pricing refresh without a reload - include maxContextTokens in ChatForm's stabilized conversation memo so the gauge reflects a changed context-window setting immediately - feed the live output estimate from the legacy content path (direct and assistants streams), setting from cumulative part text rather than accumulating deltas * 🩹 fix: Resume Usage Dedup, Agent Pricing, and Partial Override Billing - fold usage events idempotently by (runId, seq) so resume backfill no longer resets the conversation totals — a mid-stream reconnect keeps the usage of prompts already completed earlier in the session - tap replayed pending message/reasoning/content events so output streamed past the resume snapshot reaches the live estimate, not just the message - resolve cost against the agent's backing endpoint (Agents conversations report endpoint `agents` / provider `openAI`, neither of which keys a custom endpoint's tokenConfig) - getMultiplier/getCacheMultiplier fall back to the standard tables for models absent from a partial endpointTokenConfig, so a partial static override no longer bills non-listed models at defaultRate while the UI shows the correct pattern rate * 🩹 fix: Repaired Output in Gauge, Cache-Rate Keys, Config Gate, Usage Cleanup - live/completed gauge counts the repaired completion (normalized output), so under-reporting providers don't drop the response from used context - translate static tokenConfig cacheWrite/cacheRead onto the write/read keys getCacheMultiplier reads, so cache tokens bill at the configured rate instead of the prompt-rate fallback - clear the token index and usage atoms when leaving a conversation, so visited histories don't accumulate in memory for the tab's lifetime - wait for startupConfig before mounting the gauge, so a deployment with contextUsage disabled never briefly mounts it or fires the token-config query on first load * 🩹 fix: Move Token-Config Resolution to TS; Key Live Usage by Created Convo - extract the token-config resolution (override gathering + cache lookup + buildTokenConfigMap) into resolveTokenConfigMap in packages/api, leaving the /api controller a thin request-scoped wrapper (CLAUDE.md TS rule) - getConvoKey prefers the user message's real conversationId once the `created` event stamps it, so a new chat's first-response live gauge and totals land under the id TokenUsage subscribes to instead of NEW_CONVO * 🩹 fix: Clear Stale Redis Job Usage; Live-Tap Legacy Streams; Share Fetched Config - DEL the Redis job hash before re-creating it so a reused streamId can't inherit a prior run's contextUsage/tokenUsage and backfill stale usage - tap the legacy {message,text} stream branch (non-agent OpenAI/Anthropic streams) into the live estimate, not just the content path - copy a deduped fetch's token config to every sibling endpoint sharing the baseURL/key/headers, so /token-config resolves each by its own name * ⏪ revert: Don't DEL Redis job hash in createJob (breaks cross-replica resume) createJob is an idempotent join — a second replica calls it for the same streamId to share an in-flight stream's state. DELeting the hash wiped the prior replica's persisted created/usage state, so a joining replica missed the created event (GenerationJobManager cross-replica integration test). Reverts the F1 change from 2bfce0c; the stale-usage concern doesn't arise in practice (streamId is unique per generation). * 🩹 fix: Best-Effort Usage Emit; Tag Hidden Sequential-Agent Usage - wrap the ModelEndHandler usage emit in try/catch so a failed telemetry delivery (closed SSE / Redis publish error) can't abort the handler before thought-signature capture, which would break resumed tool calls - tag hidden sequential-agent usage as 'sequential' (non-primary) so the client folds it into session cost/totals but not the live context gauge, instead of letting an undefined usage_type inflate the visible gauge * 🩹 fix: Refetch Stale Token Config on Mount; Normalize Vertex for Lookup - useTokenConfigQuery refetches on mount when stale, so a user-key change that invalidates tokenConfig while the gauge is unmounted takes effect on return instead of serving the prior key's resolved config - normalize a Vertex-backed agent's provider (vertexai) to the google token-config key, so Gemini context windows and rates resolve instead of showing unknown context / $0 cost * ✨ feat: Server-Side Per-Event Cost (Authoritative Pricing for the Gauge) Move usage-cost pricing to the single source of truth. The backend prices each model call with the same billing functions (premium tiers via getMultiplier(inputTokenCount), cache rates) and emits the USD cost on on_token_usage when interface.contextCost is enabled; the client sums emitted costs instead of re-deriving from base token-config rates. - computeUsageCostUSD reuses prepareTokenSpend/prepareStructuredTokenSpend so the emitted cost matches what is billed (incl. premium thresholds) - getDefaultHandlers gains a usageCost pricing context; initialize.js wires db.getMultiplier/getCacheMultiplier gated on contextCost (agents path) - client UsageTotals carries a summed costUSD; retire the client-side rate lookups (costFromUnits/calcUsageCost) that drifted from backend pricing and produced the provider-keying / cache-key / Vertex / premium findings - keep normalizeUsageUnits for the displayed token counts; token-config is still used for the context-window meter Fixes the premium-tier session-cost under-report (gpt-5.x / gemini-3.1 above their input thresholds). * 🩹 fix: Branch-Accurate Usage Snapshot + Clearer Gauge Track Contrast - re-anchor the context snapshot from the user message to the response message at finalize. Regenerating a response branches off a shared user message, so anchoring on it made the snapshot read as "active" on both branches — switching to the sibling branch showed the wrong (other branch's) context. The response message is branch-unique, so sibling branches now correctly fall back to their own per-branch totals. - raise the gauge ring's track/fill contrast (muted track, prominent fill) so the used portion reads clearly as a fill-level indicator * 🩹 fix: Tag Sequential Usage in Billing; Emit Subagent Cost; Reset Live on Resume Errors - tag hidden sequential-agent usage `usage_type: 'sequential'` on the COLLECTED usage (not just the emit), and treat it as non-primary in recordCollectedUsage (billed, excluded from the reported output total) so hidden intermediate output stops inflating the parent's tokenCount/pruning - emit on_token_usage from the subagent usage sink (tagged `subagent`, with authoritative cost when contextCost is on) so the gauge's session cost/totals include billed subagent usage; it stays out of the live meter - call resetLive on the resumable 404 and max-retry terminal branches so the gauge doesn't keep counting stale in-flight tokens after the stream ends * 🎨 fix: Contrast the Popup Context Bar; Revert Ring Restyle - raise the popup breakdown's context progressbar contrast (muted surface-tertiary track, prominent text-primary fill) — that's the bar the contrast feedback was about - revert the gauge ring restyle (kept its original border-heavy track / text-secondary fill); the ring wasn't the element in question * 🩹 fix: Stop Snapshot Granularity Leaking Across Branches; Revert Tree Memo - a null-anchor context snapshot was treated as active on every branch, leaking one generation's granular breakdown onto sibling branches. Require a non-null (response-message) anchor on the viewed branch instead, so siblings without a matching snapshot fall back to their own totals. - revert the buildTree WeakMap memo in messages.ts. buildTree is pure (builds from shallow copies) so the memo was behaviorally identical, but it was the feature's only change to core branch-navigation selectors — removing it matches upstream and rules it out of branch-navigation debugging. * 🪙 fix: Thread Endpoint Token Config to Agent Billing, Cost, and Context Limits Custom-endpoint agents resolve an endpointTokenConfig during agent init but it never reached the AgentClient, so spending, emitted cost, and runtime max-token resolution all fell back to default rates for those agents. - Surface options.endpointTokenConfig on the returned InitializedAgent. - Pass it to the AgentClient (this.options.endpointTokenConfig) so the spending path bills at configured rates. - Thread it through usageCost to computeUsageCostUSD so emitted per-event cost matches billing. - getModelMaxTokens/getModelMaxOutputTokens fall back to the built-in map for models absent from a partial override (matches buildTokenConfigMap); consolidates the duplicated fallback in pricing.ts. * 🪙 fix: Preserve Granular Breakdown Across Branch Switches The granular context breakdown lives only in the live on_context_usage snapshot — a single per-conversation slot, anchored to the latest response and overwritten by each generation. Switching to a branch generated earlier this session lost its tool/skill/system rows and fell back to coarse totals. Retain each generation's finalized snapshot in a per-conversation map keyed by its branch-unique response id (snapshotsByAnchorFamily). When the live snapshot is off the viewed branch, walk the branch tail for its deepest stored anchor and render that breakdown. Bounded by generation count and cleared on conversation switch; the live/just-generated path is unchanged. * 🪙 fix: Harden Resume Seeding and Subagent Usage Emission - useResumableSSE: skip the trailing-output live seed when the resume carries a context snapshot; the snapshot's messageTokens already counts produced output, so seeding it again inflated usage until the next reset. - AgentClient subagent emitter: await GenerationJobManager.emitChunk like every other caller (it persists before publishing), so a floating promise can't race job cleanup and a Redis/publish failure is caught by the emitter's try/catch instead of surfacing as an unhandled rejection. * 🧪 test: Playwright Coverage for Context Breakdown Granularity Add a test-only data-testid distinguishing the granular snapshot breakdown (context-breakdown) from the coarse message-history estimate (context-estimate), then assert granularity in the mock e2e harness: - renders the granular breakdown from the live on_context_usage snapshot (guards that the snapshot event actually reaches the popover, not just the usage totals). - preserves the granular breakdown after switching branches — regenerate to overwrite the single live snapshot, switch back, and confirm the rows survive via the per-anchor snapshot history map. Branch regenerate/sibling selectors mirror the existing chat.spec branch test. All three usage specs pass against the mock pipeline. * 🪙 fix: Correct Resume Live-Seed, Fallback Re-index, and Subagent Emit Flush Codex round on the prior commit: - countTrailingOutputChars now counts only output at the very END of the aggregated content (0 when the model paused at a tool call), and the resume path always seeds it. The earlier skip-trailing-tool-parts behavior plus the skip-seed-when-snapshot gate together over- or under-counted in-flight output on resume; one rule fixes both — pre-invoke snapshot budget is never double-counted, and genuine in-flight output is no longer dropped. - useTokenUsage re-indexes from the messages cache on tail change while submitting. The cache subscriber is muted during streaming, so without a context snapshot (non-agent streams) sumBranch missed the created tail and dropped history + prompt until finalize. Bounded — tailId only shifts on created/finalize/branch-switch. - AgentClient tracks subagent usage emit promises and flushes them in chatCompletion's finally. The sink fires the emitter without awaiting, and resume reads the usage emitChunk persists (HSET), so cleanup must not race it or resumed clients miss billed subagent usage.
Summary
I implemented real-time context window and token usage tracking: a gauge beside the chat input shows tokens used vs. max context for the currently viewed message branch, with a hover breakdown of context composition (messages, system prompt, tool schemas, summary, free space), provider-reported input/output/cache totals, and opt-in session cost. The design mirrors the backend's existing token accounting instead of re-deriving it client-side, with three layered data sources that degrade gracefully: live per-model-call context snapshots, per-call usage events, and per-message
tokenCounthistory with a server-resolved limits lookup.on_token_usageSSE events fromModelEndHandlerper model call, carrying provider-reportedinput_tokens/output_tokensand cache details alongside the existingcollectedUsagerecording.on_context_usagesnapshots (post-pruneTokenBudgetBreakdownwith effective budget, instruction overhead, and remaining context) emitted by@librechat/agentsbefore each model call; the handler registration is guarded so it no-ops until@librechat/agents@3.2.35ships and activates automatically once it does.delete responseMessage.tokenCountinBaseClientso final events carry output token counts, and addedtokenCounttotMessageSchema.GET /api/endpoints/token-config: server-side resolution of context windows per configured model (including custom-endpointtokenConfigoverrides via a shared cache-key helper), with pricing rates included only when enabled — pattern matching against the pricing tables stays in one place.interface.contextUsage(default on) to gate the gauge andinterface.contextCost(default off) to gate cost display and pricing exposure.contextMeta.calibrationRatiopersisted on messages and corrected by provider-reported usage.RedisJobStoredeserialization), surfaced both throughResumeState, rebuilt client totals via backfill at sync, and re-seeded the live estimate from trailing aggregated content so revisiting an in-progress chat stays accurate.buildTreeby message-array identity inselectActiveBranchTail/getMessageBranchSiblingParentIdsso multiple cache selectors share one tree build per snapshot.usage_metadataon a final chunk (OpenAI streaming pattern), exercising the full usage pipeline in mock runs.Reuses the gauge ring and hover-card visual approach explored in #10958, rebuilt on branch-aware accounting with live streaming updates.
Dependency: full context-snapshot granularity requires
@librechat/agents@3.2.35(addsON_CONTEXT_USAGE; branch ready in the agents repo). Until then the indicator runs on usage events and per-message token history.Change Type
Testing
api: 349 tests pass (server/controllers/agents+BaseClient), including a new integration spec that drives the realRun.create→ graph → ToolNode pipeline with a usage-emitting fake model across a two-call tool loop, asserting event payloads, ordering,collectedUsageparity, and resume persistence throughGenerationJobManager.packages/api: 578 endpoint tests + 36 stream unit tests pass, including new specs forbuildTokenConfigMapresolution and usage/context resume persistence.packages/data-provider: 1212 tests pass;packages/data-schemas: 1616 tests pass.client: token index/branch-walk/cost unit tests (15) plus message-hook suites pass;tsc --noEmitclean.Test Configuration:
E2E_BASE_URL=http://localhost:3334 MOCK_LLM_PORT=8899 npm run e2e:mock).@librechat/agents@3.2.35build; they self-skip on the published3.2.33.Checklist