feat(webapp): per-client database pool metrics that survive the driver adapter - #4541
Merged
Conversation
…r adapter Report Prisma pool and query metrics for every configured client (control-plane writer/replica, run-ops writer/replica, legacy writer/replica) instead of only the control-plane writer, tagged with db_client and db_driver attributes. Pool figures come from the authoritative source per driver: pg.Pool (totalCount/idleCount/waitingCount + connect/remove counters) for driver-adapter clients, and the Rust engine metrics for quaint clients. Query counters and duration histograms come from prisma metrics for both. Adds a db.pool.connections.waiting gauge. Stops exporting Prisma metrics from the Prometheus /metrics route; pool observability now lives entirely in the OTel pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
|
Contributor
Observability mapAs of 18/100 over 413 measured of 429 entry points (base 18, no change) What this PR changed FIX FIRST
AUDIT 3 of 50 sensitive mutations record an actor. 47 without one. What the score is made ofThe score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md. |
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe change adds shared database metric registration and normalization. Standard Prisma and driver-adapter clients register pool, connection, query, and histogram sources. The tracer collects metrics for every registered client and records client, driver, pool, and waiting-connection observations. The metrics route now serves 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…tribute Label each pool with its full datasource role (control-plane-writer, control-plane-replica, run-ops-writer, run-ops-replica, legacy-run-ops-writer, legacy-run-ops-replica) instead of the generic writer/reader, matching the db.datasource span attribute so metrics and traces correlate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
ericallam
marked this pull request as ready for review
August 9, 2026 16:39
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Back the metrics-source registry with singleton() keyed by clientType, matching the app's other process-wide registries and deduping so a re-evaluated module or a repeated label registers once. Document the removal of prisma_* from the Prometheus /metrics endpoint in the server-changes note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
… note Run oxfmt on databaseMetrics.server.ts (code-quality check). Reword the server-changes note to a single user-facing sentence with no infra names, per the release-note guidance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
A failed $metrics.json() previously zero-filled every engine-derived value, so the batch observer reported spurious zeros — including for cumulative counters (db.client.queries.total, db.datasource.queries.total, db.pool.connections.opened/closed.total), which reads as a counter reset and corrupts rate math. Restore the old fail-quiet behavior per client: the collector now signals engineMetricsAvailable and leaves engine-derived fields (counters/gauges/histograms, and quaint pool figures) undefined on failure, and the tracer skips those instruments for that client. Adapter clients still emit live pg.Pool figures because those don't depend on $metrics. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KHjfL7qXHia5DTnxi1RePS
0ski
approved these changes
Aug 10, 2026
Merged
ericallam
pushed a commit
that referenced
this pull request
Aug 13, 2026
## Summary 4 new features, 24 improvements, 10 bug fixes. ## Highlights - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](#4561)) ## Improvements - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](#4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](#4551)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](#4525)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](#4516)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - The dashboard agent now has a monthly message allowance and plan-based limits on watches. Queries stay read-only with clearer errors when busy, and messages with unusual characters no longer fail to send. ([#4516](#4516)) - Meet the dashboard agent: a chat in every environment that answers questions about your runs, queues, errors and health with real data and links, replacing Ask AI everywhere it used to appear. Investigate a failed run, an error, a backed-up queue or a run that hasn't started to get a worked-through answer — what happened, why, and how to fix it, with every claim linked to the runs, errors and deploys behind it. It reads your data read-only, works on preview and dev branches with that branch's own data, and reads the same everywhere — dashboard, terminal, editor. A very long chat keeps working: the agent summarises the earlier part and carries on. **Watch…** on a run, queue, error or the health report tells you when things change: a run finishes, a queue clears or grows past a number you pick, an error comes back, an environment recovers. The answer arrives in the chat and, if you want, by email, Slack or webhook — and the agent can look into bad news on its own. A watch reaches you on any browser you sign in from, without opening the chat first. A sample of conversations is scored automatically so the agent keeps getting better; only the score and a one-line summary are kept, never your messages, data or code, and we can switch it off for your organization on request. Ask the agent instead of the Docs buttons in page headers — they stay there when the agent isn't available to you. Separately, a queue's wait times, peak depth, throughput and throttling can now be read from the API. ([#4418](#4418)) - Add backend support for delaying cron schedules within a specified window with a minimum of 60 seconds. ([#4566](#4566)) - Reduced recurring background database load from the billing-limit recovery check, so paused environments are reconciled with less overhead. ([#4590](#4590)) - Validating a schedule when deploying or updating a schedule now does less work on projects with many preview branches, so those operations stay fast as branches accumulate. ([#4598](#4598)) - Project pages now load faster for projects with a large number of preview branches, by no longer loading archived branch environments that aren't shown. ([#4595](#4595)) - Database queries that filter on a list of values now reuse cached query plans more consistently, instead of forcing the database to re-plan whenever the list length changes. ([#4480](#4480)) - Routine cleanup of old dashboard agent data now runs on its own schedule. ([#4599](#4599)) - Database connection metrics are now reported for every configured database connection instead of only the primary one, and stay accurate regardless of connection type. ([#4541](#4541)) - Deployment-related API endpoints now draw from their own generous rate limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment variables, so runtime API traffic no longer competes with deployments for the same per-environment budget. ([#4565](#4565)) - Deleting or editing a secret environment variable is now fast and no longer slows down as a project accumulates variables. ([#4555](#4555)) - Speed up personal access token lookups by indexing them on their owner ([#4588](#4588)) - Switching project or organization in the sidebar now keeps you on the same page instead of sending you back to Tasks. Pages for a specific run, deploy or other single item open the matching list instead. ([#4585](#4585)) - Reduced database load when loading the dashboard by removing an unused organization member count that was being calculated on every page navigation. ([#4587](#4587)) - The environment variables page now loads a page at a time, keeping it fast for projects with a large number of variables. Search matches variable names across every page. ([#4597](#4597)) - Groundwork for an alternative database connection driver, gated behind configuration and disabled by default, so there is no change to default behavior. ([#4539](#4539)) - Deleting an alert channel is now fast and no longer slows down as a project builds up alert history. ([#4554](#4554)) - Reduced internal overhead on the API under high load. ([#4532](#4532)) - Out-of-date upgrade prompts no longer appear in the dashboard: the "V4" badges and the notices saying preview branches and the queues table need V4 have been removed. The side menu still warns you when a project is on v3, with updated wording and a link to the v4 upgrade guide. ([#4589](#4589)) - Make background worker registration cheaper for projects with many scheduled tasks by scoping declarative schedule reconciliation to the current environment and dropping redundant schedule lookups. ([#4577](#4577)) - Speed up setting and importing environment variables for projects with many variables. ([#4579](#4579)) - Loading the deployments list is now faster, especially when filtering by deployment status on projects with many deployments. ([#4591](#4591)) - Fixed the billing limits page timing out for organizations with many preview branches, especially while a spend limit was being enforced. The page now loads quickly, so you can raise or resolve your limit without delay. ([#4594](#4594)) - Fix the Concurrency page showing the plan's default concurrency for the dev environment instead of the environment's actual limit. ([#4596](#4596)) - Creating an organization sometimes left you back on the creation form even though the organization had already been created, so clicking Create again made a duplicate. Creating an organization now completes and takes you to your new organization. ([#4530](#4530)) - Ensure creating a project completes instead of returning to its creation form after a navigation error. ([#4584](#4584)) - Renaming a project now keeps you on the project settings page and tells you what happened, instead of silently moving you to the tasks page or clearing the form with no explanation. ([#4601](#4601)) - Fixed support threads showing no account details for some customers, so the team can see your plan, organizations and projects when you get in touch. ([#4575](#4575)) - In the light theme, the Format, Clear and Copy buttons on the query editor no longer blend into the query text behind them. ([#4592](#4592)) - The health report now says start latency is "unknown" when there is no data for it, instead of showing a healthy-looking 0ms ([#4544](#4544)) - Realtime streams written inside a chat session run now use the same backend as the session itself, and runs are no longer created against a backend that cannot serve them. ([#4564](#4564)) - The grouped "watch updates" notification now shows the total number of results waiting, instead of only the most recent batch's count. ([#4525](#4525)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## trigger.dev@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](#4561)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](#4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](#4551)) - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/build@4.5.11` - `@trigger.dev/schema-to-json@4.5.11` ## @trigger.dev/core@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](#4525)) ## @trigger.dev/python@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/sdk@4.5.11` - `@trigger.dev/build@4.5.11` ## @trigger.dev/react-hooks@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/redis-worker@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/rsc@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/schema-to-json@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/sdk@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](#4516)) - Updated dependencies: - `@trigger.dev/core@4.5.11` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's
$metrics, which is owned by the Rust engine'squaintpool. Under the adapter,pg.Poolowns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton).This PR makes database metrics driver-agnostic and per-client:
db_clientanddb_driver(quaint|pg-adapter) attributes.db_clientuses our canonical datasource-role labels (control-plane-writer,control-plane-replica,run-ops-writer,run-ops-replica,legacy-run-ops-writer,legacy-run-ops-replica) — the same strings used for thedb.datasourcespan attribute, so a metric and a trace point at the same pool.pg.Pool(totalCount/idleCount/waitingCount, plus cumulative opened/closed fromconnect/removeevents).$metricspool gauges/counters, exactly as before.$metricsfor both drivers (the Rust engine executes queries in both cases).db.pool.connections.waitinggauge (pg.Pool exposes this; quaint reports 0)./metricsroute. Pool observability now lives entirely in the OTel pipeline, per driver, per client.Why
So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension.
Testing
Unit (
apps/webapp/app/utils/databaseMetrics.server.test.ts): the pure normalizer — quaint reads pool from$metrics; adapter reads pool frompg.Pooland keeps engine query metrics;busynever goes negative; graceful zeroing when$metricsis unavailable (adapter still reports live pool figures).Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series:
Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from
pg.Pool; and query counters/duration histograms keep incrementing under the pg adapter. Also verified/metrics(Prometheus) now returns zeroprisma_*series while still serving the app's own metrics.pnpm run typecheck --filter webapppasses.Notes
/metrics(Prometheus) no longer includesprisma_*series. Anything scraping that endpoint for Prisma metrics should read the equivalentdb.*metrics from the OTel exporter instead.?schema=gotcha (separate from this PR, worth flagging for rollout): since feat(webapp,database): opt-in per-client Prisma driver adapters #4539 parses?schema=from the DSN and passes{ schema }to the adapter, node-postgres sendssearch_pathas a startup parameter. A transaction-mode PgBouncer rejects that withFATAL: unsupported startup parameter: search_path. Our prod control-plane DSNs use the defaultpublicschema with no?schema=param, so this is latent, but any client we flip to the adapter must not carry?schema=in its DSN (or the pooler needsignore_startup_parameters = search_path).