feat(webapp,database): opt-in per-client Prisma driver adapters - #4539
Conversation
Add per-client env vars to route each Prisma client through @prisma/adapter-pg (node-postgres) instead of the built-in engine driver. All default off, so behavior is unchanged unless a flag is set: - CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER - CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER - RUN_OPS_DATABASE_WRITER_DRIVER_ADAPTER - RUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTER - RUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTER - RUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTER Enables the driverAdapters preview feature on both schemas (keeps the Rust query engine; does not add queryCompiler). Each adapter pool is built with a bounded connectionTimeoutMillis and an onPoolError handler. Handle the connect-failure differences the adapter introduces: - isInfrastructureError now recognizes the adapter's connect-failure shapes (P2010 'not reachable' and raw ECONNREFUSED/ENOTFOUND-class errors) so the DB host is still scrubbed from API-client errors and infra failures are logged. - isPrismaRetriableError treats the adapter pool-acquire timeout as retriable, preserving the P2024 retry behavior. refs TRI-13039 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe web application adds optional PostgreSQL driver-adapter support for control-plane, run-ops, and legacy run-ops Prisma writers and replicas. Environment flags independently enable each adapter. Adapter-backed clients use configured PostgreSQL pools. Default behavior continues to use datasource URLs. Prisma generators and package dependencies are updated. Connectivity detection handles adapter timeout messages, network errors, and connectivity-related 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- Pass the per-client resolved connection limit into the adapter pool instead of always using DATABASE_CONNECTION_LIMIT, so per-client overrides (e.g. RUN_OPS_DATABASE_READ_REPLICA_CONNECTION_LIMIT) are honored on the adapter path. - Build the adapter pool from the base DSN (drops prisma-only URL params the pg driver ignores and the duplicate application_name). - Scope the connectivity message match to 'database not reachable' so a generic 'not reachable' error is no longer misclassified as infrastructure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Pass the datasource schema (?schema=) to PrismaPg as its {schema} option so
custom-schema installs keep talking to the right schema on the adapter (the
adapter does not honor ?schema= in the connection string).
- Set disposeExternalPool: true so $disconnect() closes the pg pool instead of
leaking sockets, matching the engine-driver path.
- Guard the two $metrics consumers (the /metrics route and the OTel batch
observable callback) so a client on a driver adapter degrades to empty metrics
instead of failing the scrape / rejecting the callback.
- isPrismaRetriableError checks the adapter acquire-timeout message independently
of the coded-error branch, so the pool-acquire retry still engages if the
timeout arrives wrapped as a coded error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Observability mapAs of 18/100 over 413 measured of 429 entry points (base 18, no change) What this PR changed FIX FIRST
AUDIT 3 of 50 sensitive mutations record an actor. 47 without one. What the score is made ofThe score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md. |
The retry decision used retryCodes.includes(error.code) directly, inside the isPrismaKnownError branch, so the broadened isPrismaRetriableError check never governed retries. Route the retry decision through isPrismaRetriableError so the adapter's pool-acquire timeout is retried like P2024 was, while keeping prismaError()/swallow behavior for coded errors only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r drainer isRetryablePgError only recognized the Rust engine's DB-unreachable shapes (P1001 / "Can't reach database server"). Under a driver adapter the same outage surfaces as P2010 "Database not reachable" / ECONNREFUSED / ENOTFOUND, so buffered runs were permanently failed on a transient outage. Reuse the shared looksLikeConnectivityError predicate (now exported from prismaErrors) so those are retried too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r adapter (#4541) ## What Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool. Under the adapter, `pg.Pool` owns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton). This PR makes database metrics driver-agnostic and per-client: - Every configured client registers a metrics source: control-plane writer/replica, run-ops writer/replica, legacy writer/replica. Previously only the control-plane writer singleton was scraped. - Each OTel instrument is observed per client with `db_client` and `db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our canonical datasource-role labels (`control-plane-writer`, `control-plane-replica`, `run-ops-writer`, `run-ops-replica`, `legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings used for the `db.datasource` span attribute, so a metric and a trace point at the same pool. - Pool figures come from the authoritative source per driver: - **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`, plus cumulative opened/closed from `connect`/`remove` events). - **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly as before. - Query counters and duration histograms still come from `$metrics` for both drivers (the Rust engine executes queries in both cases). - New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint reports 0). - Stops exporting Prisma metrics from the Prometheus `/metrics` route. Pool observability now lives entirely in the OTel pipeline, per driver, per client. ## Why So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension. ## Testing Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure normalizer — quaint reads pool from `$metrics`; adapter reads pool from `pg.Pool` and keeps engine query metrics; `busy` never goes negative; graceful zeroing when `$metrics` is unavailable (adapter still reports live pool figures). Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series: ``` db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1 db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing ``` Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from `pg.Pool`; and query counters/duration histograms keep incrementing under the pg adapter. Also verified `/metrics` (Prometheus) now returns zero `prisma_*` series while still serving the app's own metrics. `pnpm run typecheck --filter webapp` passes. ## Notes - `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything scraping that endpoint for Prisma metrics should read the equivalent `db.*` metrics from the OTel exporter instead. - **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging for rollout):** since #4539 parses `?schema=` from the DSN and passes `{ schema }` to the adapter, node-postgres sends `search_path` as a startup parameter. A transaction-mode PgBouncer rejects that with `FATAL: unsupported startup parameter: search_path`. Our prod control-plane DSNs use the default `public` schema with no `?schema=` param, so this is latent, but any client we flip to the adapter must not carry `?schema=` in its DSN (or the pooler needs `ignore_startup_parameters = search_path`). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary 4 new features, 24 improvements, 10 bug fixes. ## Highlights - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](#4561)) ## Improvements - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](#4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](#4551)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](#4525)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](#4516)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - The dashboard agent now has a monthly message allowance and plan-based limits on watches. Queries stay read-only with clearer errors when busy, and messages with unusual characters no longer fail to send. ([#4516](#4516)) - Meet the dashboard agent: a chat in every environment that answers questions about your runs, queues, errors and health with real data and links, replacing Ask AI everywhere it used to appear. Investigate a failed run, an error, a backed-up queue or a run that hasn't started to get a worked-through answer — what happened, why, and how to fix it, with every claim linked to the runs, errors and deploys behind it. It reads your data read-only, works on preview and dev branches with that branch's own data, and reads the same everywhere — dashboard, terminal, editor. A very long chat keeps working: the agent summarises the earlier part and carries on. **Watch…** on a run, queue, error or the health report tells you when things change: a run finishes, a queue clears or grows past a number you pick, an error comes back, an environment recovers. The answer arrives in the chat and, if you want, by email, Slack or webhook — and the agent can look into bad news on its own. A watch reaches you on any browser you sign in from, without opening the chat first. A sample of conversations is scored automatically so the agent keeps getting better; only the score and a one-line summary are kept, never your messages, data or code, and we can switch it off for your organization on request. Ask the agent instead of the Docs buttons in page headers — they stay there when the agent isn't available to you. Separately, a queue's wait times, peak depth, throughput and throttling can now be read from the API. ([#4418](#4418)) - Add backend support for delaying cron schedules within a specified window with a minimum of 60 seconds. ([#4566](#4566)) - Reduced recurring background database load from the billing-limit recovery check, so paused environments are reconciled with less overhead. ([#4590](#4590)) - Validating a schedule when deploying or updating a schedule now does less work on projects with many preview branches, so those operations stay fast as branches accumulate. ([#4598](#4598)) - Project pages now load faster for projects with a large number of preview branches, by no longer loading archived branch environments that aren't shown. ([#4595](#4595)) - Database queries that filter on a list of values now reuse cached query plans more consistently, instead of forcing the database to re-plan whenever the list length changes. ([#4480](#4480)) - Routine cleanup of old dashboard agent data now runs on its own schedule. ([#4599](#4599)) - Database connection metrics are now reported for every configured database connection instead of only the primary one, and stay accurate regardless of connection type. ([#4541](#4541)) - Deployment-related API endpoints now draw from their own generous rate limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment variables, so runtime API traffic no longer competes with deployments for the same per-environment budget. ([#4565](#4565)) - Deleting or editing a secret environment variable is now fast and no longer slows down as a project accumulates variables. ([#4555](#4555)) - Speed up personal access token lookups by indexing them on their owner ([#4588](#4588)) - Switching project or organization in the sidebar now keeps you on the same page instead of sending you back to Tasks. Pages for a specific run, deploy or other single item open the matching list instead. ([#4585](#4585)) - Reduced database load when loading the dashboard by removing an unused organization member count that was being calculated on every page navigation. ([#4587](#4587)) - The environment variables page now loads a page at a time, keeping it fast for projects with a large number of variables. Search matches variable names across every page. ([#4597](#4597)) - Groundwork for an alternative database connection driver, gated behind configuration and disabled by default, so there is no change to default behavior. ([#4539](#4539)) - Deleting an alert channel is now fast and no longer slows down as a project builds up alert history. ([#4554](#4554)) - Reduced internal overhead on the API under high load. ([#4532](#4532)) - Out-of-date upgrade prompts no longer appear in the dashboard: the "V4" badges and the notices saying preview branches and the queues table need V4 have been removed. The side menu still warns you when a project is on v3, with updated wording and a link to the v4 upgrade guide. ([#4589](#4589)) - Make background worker registration cheaper for projects with many scheduled tasks by scoping declarative schedule reconciliation to the current environment and dropping redundant schedule lookups. ([#4577](#4577)) - Speed up setting and importing environment variables for projects with many variables. ([#4579](#4579)) - Loading the deployments list is now faster, especially when filtering by deployment status on projects with many deployments. ([#4591](#4591)) - Fixed the billing limits page timing out for organizations with many preview branches, especially while a spend limit was being enforced. The page now loads quickly, so you can raise or resolve your limit without delay. ([#4594](#4594)) - Fix the Concurrency page showing the plan's default concurrency for the dev environment instead of the environment's actual limit. ([#4596](#4596)) - Creating an organization sometimes left you back on the creation form even though the organization had already been created, so clicking Create again made a duplicate. Creating an organization now completes and takes you to your new organization. ([#4530](#4530)) - Ensure creating a project completes instead of returning to its creation form after a navigation error. ([#4584](#4584)) - Renaming a project now keeps you on the project settings page and tells you what happened, instead of silently moving you to the tasks page or clearing the form with no explanation. ([#4601](#4601)) - Fixed support threads showing no account details for some customers, so the team can see your plan, organizations and projects when you get in touch. ([#4575](#4575)) - In the light theme, the Format, Clear and Copy buttons on the query editor no longer blend into the query text behind them. ([#4592](#4592)) - The health report now says start latency is "unknown" when there is no data for it, instead of showing a healthy-looking 0ms ([#4544](#4544)) - Realtime streams written inside a chat session run now use the same backend as the session itself, and runs are no longer created against a backend that cannot serve them. ([#4564](#4564)) - The grouped "watch updates" notification now shows the total number of results waiting, instead of only the most recent batch's count. ([#4525](#4525)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## trigger.dev@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](#4561)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](#4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](#4551)) - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/build@4.5.11` - `@trigger.dev/schema-to-json@4.5.11` ## @trigger.dev/core@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](#4525)) ## @trigger.dev/python@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/sdk@4.5.11` - `@trigger.dev/build@4.5.11` ## @trigger.dev/react-hooks@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/redis-worker@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/rsc@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/schema-to-json@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/sdk@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](#4516)) - Updated dependencies: - `@trigger.dev/core@4.5.11` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
What
Adds an opt-in path to run each Prisma client through
@prisma/adapter-pg(the node-postgres driver) instead of the built-in engine driver, controlled by a per-client env var, all off by default:CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTERCONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTERRUN_OPS_DATABASE_WRITER_DRIVER_ADAPTERRUN_OPS_DATABASE_REPLICA_DRIVER_ADAPTERRUN_OPS_LEGACY_DATABASE_WRITER_DRIVER_ADAPTERRUN_OPS_LEGACY_DATABASE_REPLICA_DRIVER_ADAPTERWith every flag unset the construction path is byte-identical to today (
datasourcesURL + Rust engine), so this is inert until a flag is turned on. Per-client granularity allows enabling the adapter only where it's wanted.How
driverAdapterspreview feature on both schemas (@trigger.dev/databaseand@internal/run-ops-database). This keeps the Rust query engine — it does NOT addqueryCompiler— so query behavior, result types, and engine tracing spans are unchanged.buildDriverAdapterPoolbuilds each client'spg.Poolwith an explicitmax, a boundedconnectionTimeoutMillis(the node-postgres pool otherwise waits unbounded on acquire), and anonPoolErrorhandler (an unhandled idle-connection error would otherwise crash the process). Threaded through all four client builders via auseDriverAdapterflag.@prisma/adapter-pg+@types/pgto the webapp;pgis already pinned at8.15.6(adapter-pg 6.x requirespg < 8.17).Connect-failure handling (the important correctness/security bit)
Under the adapter an unreachable DB no longer surfaces as
PrismaClientInitializationError/P1001; it becomes aP2010"Database not reachable: " (or a rawECONNREFUSED/ENOTFOUND-class error). Two handlers are updated so a client on the adapter behaves like today:isInfrastructureErrornow recognizes those shapes (P2010 with a connectivity message, and raw connectivity errno codes). Without this, the DB hostname would leak into API-client-facing errors and the failure would go unlogged. Security-relevant.isPrismaRetriableErrortreats the adapter's pool-acquire timeout ("timeout exceeded when trying to connect") as retriable, preserving theP2024retry behavior the adapter otherwise drops.Evidence
Validated on an isolated stack that mirrors the production DB topology (chained PgBouncers in front of writer + reader):
metaare byte-identical between the engine driver and the adapter across the queried shapes (unique-constraintmeta.target, record-not-found, transaction-timeout, serialization-failure, etc.).Rollout / rollback
All flags default off; enable per client via env var, roll back by unsetting and redeploying (no data migration). Recommended first target is a single writer; enable one client at a time.
Follow-ups (not in this PR)
$metrics-based pool observability is removed under the adapter (the Prometheus route +db.pool.connections.*instruments); the metrics replacement (viapg.Poolcounters) lands in a separate PR.maxWaitdoes not bound pool acquisition —connectionTimeoutMillisdoes.Note on connection-string parameters
The adapter pool is built from the base DSN, so Prisma-specific DSN parameters that node-postgres does not understand are not honored when a client is on the adapter:
sslaccept,sslcert, etc.) — node-postgres usessslmode/sslinstead. Our production DSNs do not use these Prisma-specific TLS params, but any deployment whose DSN relies on them must be checked before enabling a flag.pgbouncer=trueandstatement_cache_size— effectively moot under the adapter, which uses no persistent named prepared statements.connection_limit,pool_timeout, andschemaare handled explicitly (passed asmax/connectionTimeoutMillisand PrismaPg's{schema}option).refs TRI-13039