fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies - #4532

ericallam · 2026-08-07T13:49:58Z

What

The internal tracing ParentBasedSampler in tracer.server.ts left remoteParentSampled at its default of AlwaysOn. Any request arriving with a traceparent whose sampled flag was set got recorded in full, bypassing INTERNAL_OTEL_TRACE_SAMPLING_RATE entirely. Because the SDK propagates its (always-sampled) trace context on calls back to the platform from inside running tasks, the large majority of API server spans inherited a sampled parent and ignored the divisor. The sampling knob was effectively inert on the busiest service.

This registers a custom propagator (NonInheritingTraceContextPropagator) that stops adopting the inbound trace as the parent:

inject still delegates to the standard W3C trace + baggage propagators, so outbound propagation is unchanged.
extract drops the parent span (trace.deleteSpan) while preserving baggage, so every incoming request roots its own trace and the ratio sampler applies uniformly.

remoteParentSampled is also set to the ratio sampler as a belt-and-suspenders fallback, in case an inbound sampled parent ever reaches the sampler another way.

Two effects: the divisor becomes effective on the API server, and the API no longer stitches onto (and inflates) the propagated task-run traces, which is where the very large, un-thinnable trace chains came from. Rooting each request removes those chains rather than only diluting them.

Only the internal APM trace pipeline (INTERNAL_OTEL_TRACE_EXPORTER_URL) is affected. The user-facing run-trace pipeline (otel.v1.traces -> ClickHouse) is a separate path and is untouched. The only consumer of the global propagator's extract is the OTel HTTP/Express auto-instrumentation, so the blast radius is inbound-request trace shape.

Evidence (local full-stack red/green, divisor 10)

A local OTLP/JSON sink counting spans; a driver fires N requests at a real endpoint, each carrying a distinct sampled traceparent, then counts how many spans/traces carry that run's marker.

run	code	sent	kept traces	kept fraction
before	unmodified	500	500	1.00
after	this PR	500	67	0.134
after	this PR	2000	213	0.1065

Before: 100% of inherited-sampled requests kept, divisor ignored. After: ~10% kept (the divisor), converging on it at larger N. In every after-run each kept request is a single self-rooted trace (kept spans == kept distinct traces), confirming the inherited chains are gone, not just thinned. typecheck passes.

Rollout / rollback

No flag. Behavior stays governed by the existing INTERNAL_OTEL_TRACE_SAMPLING_RATE. Rollback is a straight revert with no data migration.

Notes

Internal dashboards that count raw span or request volume from this pipeline will read lower once this ships. That is expected: those counts were inflated by the bypass, not a real drop in traffic. Latency/percentile monitors retain plenty of samples at the current divisor.

refs TRI-13031

…e sampling applies The internal APM tracer's ParentBasedSampler left remoteParentSampled at its AlwaysOn default, so any request arriving with a sampled traceparent (SDK task-run traces propagated in from running tasks) was recorded in full, ignoring INTERNAL_OTEL_TRACE_SAMPLING_RATE. On api that is ~99.6% of spans, so the divisor was effectively inert. Register a non-inheriting propagator: inject still delegates to W3C trace+baggage so outbound propagation is unchanged, but extract drops the parent span, so every inbound request roots its own trace and the ratio sampler applies uniformly. This also stops api stitching onto (and inflating) the SDK's task-run traces, which is where the untrimmable multi-thousand-span chains came from. Also set remoteParentSampled to the ratio sampler as a fallback.

changeset-bot · 2026-08-07T13:50:05Z

⚠️ No Changeset found

Latest commit: b9f887d

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2026-08-07T13:52:41Z

Walkthrough

The webapp tracer now uses typed composite W3C trace-context and baggage propagators. Its non-inheriting propagator removes extracted span contexts while preserving propagation behavior. Provider registration installs this propagator with the existing context manager. Sampling now reuses one ratio sampler for custom root sampling and remote sampled-parent handling. A change note documents reduced internal API overhead under high load.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly describes the main change: preventing inbound sampled traceparents from bypassing API trace sampling.
Description check	✅ Passed	The description explains the change, testing evidence, affected pipelines, rollout, rollback, and expected impact in sufficient detail.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feature/tri-13031-api-trace-sampling-is-inert-996-of-its-spans-bypass-the

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

devin-ai-integration

Devin Review found 1 potential issue.

## Summary 4 new features, 24 improvements, 10 bug fixes. ## Highlights - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](#4561)) ## Improvements - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](#4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](#4551)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](#4525)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](#4516)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - The dashboard agent now has a monthly message allowance and plan-based limits on watches. Queries stay read-only with clearer errors when busy, and messages with unusual characters no longer fail to send. ([#4516](#4516)) - Meet the dashboard agent: a chat in every environment that answers questions about your runs, queues, errors and health with real data and links, replacing Ask AI everywhere it used to appear. Investigate a failed run, an error, a backed-up queue or a run that hasn't started to get a worked-through answer — what happened, why, and how to fix it, with every claim linked to the runs, errors and deploys behind it. It reads your data read-only, works on preview and dev branches with that branch's own data, and reads the same everywhere — dashboard, terminal, editor. A very long chat keeps working: the agent summarises the earlier part and carries on. **Watch…** on a run, queue, error or the health report tells you when things change: a run finishes, a queue clears or grows past a number you pick, an error comes back, an environment recovers. The answer arrives in the chat and, if you want, by email, Slack or webhook — and the agent can look into bad news on its own. A watch reaches you on any browser you sign in from, without opening the chat first. A sample of conversations is scored automatically so the agent keeps getting better; only the score and a one-line summary are kept, never your messages, data or code, and we can switch it off for your organization on request. Ask the agent instead of the Docs buttons in page headers — they stay there when the agent isn't available to you. Separately, a queue's wait times, peak depth, throughput and throttling can now be read from the API. ([#4418](#4418)) - Add backend support for delaying cron schedules within a specified window with a minimum of 60 seconds. ([#4566](#4566)) - Reduced recurring background database load from the billing-limit recovery check, so paused environments are reconciled with less overhead. ([#4590](#4590)) - Validating a schedule when deploying or updating a schedule now does less work on projects with many preview branches, so those operations stay fast as branches accumulate. ([#4598](#4598)) - Project pages now load faster for projects with a large number of preview branches, by no longer loading archived branch environments that aren't shown. ([#4595](#4595)) - Database queries that filter on a list of values now reuse cached query plans more consistently, instead of forcing the database to re-plan whenever the list length changes. ([#4480](#4480)) - Routine cleanup of old dashboard agent data now runs on its own schedule. ([#4599](#4599)) - Database connection metrics are now reported for every configured database connection instead of only the primary one, and stay accurate regardless of connection type. ([#4541](#4541)) - Deployment-related API endpoints now draw from their own generous rate limit budget, configurable via the `DEPLOYMENT_RATE_LIMIT_*` environment variables, so runtime API traffic no longer competes with deployments for the same per-environment budget. ([#4565](#4565)) - Deleting or editing a secret environment variable is now fast and no longer slows down as a project accumulates variables. ([#4555](#4555)) - Speed up personal access token lookups by indexing them on their owner ([#4588](#4588)) - Switching project or organization in the sidebar now keeps you on the same page instead of sending you back to Tasks. Pages for a specific run, deploy or other single item open the matching list instead. ([#4585](#4585)) - Reduced database load when loading the dashboard by removing an unused organization member count that was being calculated on every page navigation. ([#4587](#4587)) - The environment variables page now loads a page at a time, keeping it fast for projects with a large number of variables. Search matches variable names across every page. ([#4597](#4597)) - Groundwork for an alternative database connection driver, gated behind configuration and disabled by default, so there is no change to default behavior. ([#4539](#4539)) - Deleting an alert channel is now fast and no longer slows down as a project builds up alert history. ([#4554](#4554)) - Reduced internal overhead on the API under high load. ([#4532](#4532)) - Out-of-date upgrade prompts no longer appear in the dashboard: the "V4" badges and the notices saying preview branches and the queues table need V4 have been removed. The side menu still warns you when a project is on v3, with updated wording and a link to the v4 upgrade guide. ([#4589](#4589)) - Make background worker registration cheaper for projects with many scheduled tasks by scoping declarative schedule reconciliation to the current environment and dropping redundant schedule lookups. ([#4577](#4577)) - Speed up setting and importing environment variables for projects with many variables. ([#4579](#4579)) - Loading the deployments list is now faster, especially when filtering by deployment status on projects with many deployments. ([#4591](#4591)) - Fixed the billing limits page timing out for organizations with many preview branches, especially while a spend limit was being enforced. The page now loads quickly, so you can raise or resolve your limit without delay. ([#4594](#4594)) - Fix the Concurrency page showing the plan's default concurrency for the dev environment instead of the environment's actual limit. ([#4596](#4596)) - Creating an organization sometimes left you back on the creation form even though the organization had already been created, so clicking Create again made a duplicate. Creating an organization now completes and takes you to your new organization. ([#4530](#4530)) - Ensure creating a project completes instead of returning to its creation form after a navigation error. ([#4584](#4584)) - Renaming a project now keeps you on the project settings page and tells you what happened, instead of silently moving you to the tasks page or clearing the form with no explanation. ([#4601](#4601)) - Fixed support threads showing no account details for some customers, so the team can see your plan, organizations and projects when you get in touch. ([#4575](#4575)) - In the light theme, the Format, Clear and Copy buttons on the query editor no longer blend into the query text behind them. ([#4592](#4592)) - The health report now says start latency is "unknown" when there is no data for it, instead of showing a healthy-looking 0ms ([#4544](#4544)) - Realtime streams written inside a chat session run now use the same backend as the session itself, and runs are no longer created against a backend that cannot serve them. ([#4564](#4564)) - The grouped "watch updates" notification now shows the total number of results waiting, instead of only the most recent batch's count. ([#4525](#4525)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## trigger.dev@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - Allow `trigger deploy` to authenticate with an environment API key from `TRIGGER_ACCESS_TOKEN`. ([#4561](#4561)) - The dev environment onboarding now tracks real progress. After you run `init`, the setup checklist marks your project as initialized, and it updates live as your dev server connects and your tasks register. The blank state also adds a "Copy AI agent prompt" button that copies a ready-to-paste setup prompt (pre-filled with your project reference) for Claude Code, Cursor, or any coding agent. ([#4563](#4563)) The `init` scaffold now imports from `@trigger.dev/sdk` instead of the deprecated `@trigger.dev/sdk/v3` subpath. - Deployed images now ship dependencies and bundled task code as separate layers. Repeat deploys with unchanged dependencies typically push and pull far less data, making deploys and worker image pulls faster. ([#4551](#4551)) - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/build@4.5.11` - `@trigger.dev/schema-to-json@4.5.11` ## @trigger.dev/core@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - The current-worker API now reports each task's queue, so you can see which tasks write to a given queue. ([#4525](#4525)) ## @trigger.dev/python@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` - `@trigger.dev/sdk@4.5.11` - `@trigger.dev/build@4.5.11` ## @trigger.dev/react-hooks@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/redis-worker@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/rsc@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/schema-to-json@4.5.11 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.11` ## @trigger.dev/sdk@4.5.11 ### Patch Changes - Chat in the browser now reconnects when the connection drops mid-turn, instead of leaving the reply stuck as if it were still generating. Reports can be fetched as structured data with the `json` format, and the shortest report period is now one minute (`1m`, `30m`, `1h`, `7d`). The `mint-token` command's help is clearer too: a token minted without `--cap` is read-only, and `--ttl` shows the correct maximum lifetime of 7 days. ([#4418](#4418)) - Watch-mode chat streams now survive quiet windows and page reloads, and a reply cut off by a lost connection shows an error instead of appearing finished. Aborting a resumed subscription only closes your local stream — call `stopGeneration(chatId)` or pass `stopOnAbort: true` to stop the run. Also fixed a race where quickly restarting a stream could break stop and reconnect, and stopping a chat now hands it back to your other tabs instead of leaving them read-only. ([#4516](#4516)) - Updated dependencies: - `@trigger.dev/core@4.5.11` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This comment was marked as resolved.

Sign in to view

ericallam marked this pull request as ready for review August 7, 2026 14:05

devin-ai-integration Bot reviewed Aug 7, 2026

View reviewed changes

Comment thread apps/webapp/app/v3/tracer.server.ts

nicktrn approved these changes Aug 7, 2026

View reviewed changes

ericallam merged commit 63176a6 into main Aug 7, 2026
45 checks passed

ericallam deleted the feature/tri-13031-api-trace-sampling-is-inert-996-of-its-spans-bypass-the branch August 7, 2026 14:13

github-actions Bot mentioned this pull request Aug 10, 2026

chore: release v4.5.11 #4557

Merged

Navigation Menu

Uh oh!

Uh oh!

fix(webapp): stop api inheriting inbound sampled traceparents so trace sampling applies - #4532