Skip to content

Migrate context bridge to the all-native DDProf API (Phase 2) - #11899

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits into
masterfrom
rkennke/profiler-all-native-context-phase2
Jul 28, 2026
Merged

Migrate context bridge to the all-native DDProf API (Phase 2)#11899
gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits into
masterfrom
rkennke/profiler-all-native-context-phase2

Conversation

@rkennke

@rkennke rkennke commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What

Migrates the profiler context bridge off the deprecated DirectByteBuffer (DBB) context API onto java-profiler's all-native API (setTraceContext / clearTraceContext / setContextValue / clearContextValue).

This is Phase 2 of the all-native context work (Phase 1 = DataDog/java-profiler#631).

Why

  • Fixes the virtual-thread use-after-free: the DBB path cached a DirectByteBuffer over the native OTEP record; on a carrier that a mounted virtual thread later migrated off, the record was freed while the cached buffer kept being written. The all-native API resolves the current carrier's record inside each JNI call — no cached buffer to dangle.
  • Per-activation perf: activation collapses from setContext + 2×setContextValue (3 JNI calls) into one setTraceContext (~28% faster on the combined cycle per the design-note benchmark).

Changes

  • DatadogProfilingIntegration.activate → one setTraceContext(...) carrying trace/span context + operation & resource attributes; close/clearContextclearTraceContext().
  • DatadogProfiler: setContextValue/clearContextValue/reapplyAppContext/syncNativeAppContext are now all-native. reapplyAppContext uses a per-slot native setContextValue loop — native setContextValue publishes the record (valid=1) even with no active span, so app context stays visible between spans (preserves Restore app context attributes wiped by ddprof setContext on span activation #11646). A native batch reapply is deferred to a measured follow-up (java-profiler PROF-15361).
  • AppContextSnapshot simplified to strings-only (the native path resolves each value's encoding via the process-wide cache — no cached id/utf8/snapshotTags DBB read).
  • snapshot() uses the new native copyContextTags read (no ThreadContext/DBB, so it observes native writes without resetting the record).
  • ContextSetter kept only for offsetOf + size (pure Java); no DBB usage remains.

Behavior notes

  • Per-slot reapply is not atomic across app attributes. reapplyAppContext publishes each app-managed attribute with its own native setContextValue (detach→attach), so a profiler sample landing mid-reapply can observe a partial app-attribute set on activation — a window the old batch fast-path (setContextValuesByIdAndBytes, single attach) did not have. This is deliberate for the expected small app-attribute cardinality (0–2); the single-call batch reapply is the measured follow-up (PROF-15361). Trace/span + operation/resource stay atomic (one setTraceContext).
  • Defensive zero-span guard. The native setTraceContext rejects spanId==0 (it is the activation path; clearing is clearTraceContext). Span ids are non-zero by construction (IdGenerationStrategy never yields 0, DDSpanId.ZERO means "no span", and DDSpanContext is the only ProfilerContext), so the bridge routes a stray zero span to clearTraceContext() rather than letting the IllegalArgumentException be swallowed over stale context.

Dependency

Requires java-profiler's all-native API, from DataDog/java-profiler#631. Master now pins ddprof = 1.47.0 (#12055), which includes it, so this branch has been rebased and builds against the published release — no local snapshot needed.

Status / verification

  • Rebased onto master with the published ddprof:1.47.0 (Bump java-profiler (ddprof) to 1.47.0 #12055); compiles clean (no cannot find symbol on the new calls) and the profiling-ddprof unit test suite is GREEN on Linux post-rebase.
  • Re-verified GREEN on Linux against the genuinely published ddprof:1.47.0 release (native lib extracted from the published jar, not a local snapshot): dd-smoke-tests:profiling-integration-tests (CodeHotspotsTest, JFRBasedProfilingIntegrationTest) — 20 tests, 2 skipped (both pre-existing/expected), 0 failures, 0 errors (end-to-end agent + profiler + JFR, exercising span/op/resource context).
  • Two correctness bugs surfaced by automated PR review (stale-value resurrection on a rejected setContextValue, and reapplyAppContext overwriting freshly-written operation/resource values) were fixed and covered by new regression assertions in DatadogProfilerTest, each confirmed to fail without its corresponding fix.

🤖 Generated with Claude Code

gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jul 23, 2026
Bump java-profiler (ddprof) to 1.47.0

Required to continue work on the all-native profiler context bridge (#11899).

Merge branch 'master' into rkennke/bump-ddprof-1.47.0

Raise agent jar size budget for java-profiler 1.47.0

The ddprof 1.47.0 native library growth pushed the shadow jar past the
previous budget; raise it to unblock the verifyAgentJarContents check.

Merge branch 'master' into rkennke/bump-ddprof-1.47.0

Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
@rkennke
rkennke force-pushed the rkennke/profiler-all-native-context-phase2 branch from b1d70ef to 07a20b2 Compare July 23, 2026 19:51
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 57.70% (-0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3668c1f | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.95 s 13.93 s [-0.5%; +0.8%] (no difference)
startup:insecure-bank:tracing:Agent 12.97 s 12.95 s [-0.4%; +0.8%] (no difference)
startup:petclinic:appsec:Agent 16.72 s 16.77 s [-1.3%; +0.7%] (no difference)
startup:petclinic:iast:Agent 16.81 s 16.89 s [-1.5%; +0.6%] (no difference)
startup:petclinic:profiling:Agent 16.70 s 16.64 s [-0.6%; +1.3%] (no difference)
startup:petclinic:sca:Agent 16.24 s 16.73 s [-7.3%; +1.6%] (no difference)
startup:petclinic:tracing:Agent 15.89 s 15.68 s [-3.0%; +5.8%] (no difference)

Commit: 3668c1fd · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@rkennke
rkennke marked this pull request as ready for review July 24, 2026 06:37
@rkennke
rkennke requested a review from a team as a code owner July 24, 2026 06:37
@dd-octo-sts

dd-octo-sts Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@dd-octo-sts dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Jul 24, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 07a20b2f98

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

Replacing an existing profiling context value with a >255-byte string now clears the native slot but retains its Java snapshot; the next span activation resurrects the stale prior value. The DBB base path retains the prior value continuously, so samples can lose and later regain incorrect context across this migration.

📊 Validated against 10 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit 07a20b2 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

rkennke added a commit that referenced this pull request Jul 24, 2026
- setContextValue: a rejected native write (e.g. >255-byte UTF-8) clears the
  native slot but left the Java snapshot untouched, so the attribute read as
  unset until the next span boundary silently resurrected the stale prior
  value. Reapply immediately on rejection so the prior value stays visible
  continuously, matching pre-migration DBB behavior.
- setTraceContext: reapplyAppContext() ran unconditionally after the native
  call, so when profiling.context.attributes also names
  _dd.trace.operation/resource (with span-name/resource-name context
  enabled), the trailing reapply clobbered the span-derived value that
  setTraceContext just wrote to the same offset with a stale app-recorded
  one. reapplyAppContext now takes the operation/resource offsets to skip.

Both were flagged independently by Codex and Datadog Autotest PR review bots
on #11899, with a concrete repro for the first. Added regression coverage to
DatadogProfilerTest#testContextRegistration for both (verified each new
assertion fails without its corresponding fix).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
rkennke and others added 5 commits July 24, 2026 17:58
Switch the profiler context bridge off the deprecated DirectByteBuffer (DBB)
context API onto java-profiler's all-native API (setTraceContext /
clearTraceContext / setContextValue / clearContextValue). This eliminates the
virtual-thread use-after-free (the DBB cached buffer that dangled on carrier
migration) and folds the per-activation sequence (setContext + two
setContextValue) into a single native call.

- DatadogProfilingIntegration.activate: one setTraceContext(...) carrying
  trace/span context + operation and resource attributes (3 JNI calls -> 1);
  close/clearContext: clearTraceContext() (wipes op/resource slots too).
- DatadogProfiler: setContextValue/clearContextValue/reapplyAppContext/
  syncNativeAppContext are now all-native. reapplyAppContext uses a per-slot
  native setContextValue loop (native setContextValue publishes valid=1, so app
  context stays visible without an active span — preserves PR #11646). A native
  batch reapply is deferred to a measured follow-up (java-profiler PROF-15361).
- AppContextSnapshot simplified to strings-only (the native path resolves each
  value's encoding via the process-wide cache; no cached id/utf8/snapshotTags).
- snapshot() uses the new native copyContextTags read (no ThreadContext/DBB, so
  it observes native writes without resetting the record).
- ContextSetter kept only for offsetOf + size (pure Java); no DBB usage remains.

Requires java-profiler with the all-native API (ddprof >= the phase-1 release;
DataDog/java-profiler#631). Build/test with -PddprofUseSnapshot=true against a
local publishToMavenLocal 1.47.0-SNAPSHOT until that ships. ddprof suite is
Linux-gated (assumeTrue(isLinux)) — verify on Linux/CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ContextValue

The native setTraceContext rejects spanId==0 with IllegalArgumentException (it is
the activation path; clearing is clearTraceContext). The bridge's catch(Throwable)
would swallow that throw and leave the previous span's context stale on the thread.
Span ids are non-zero by construction (IdGenerationStrategy never yields 0,
DDSpanId.ZERO means "no span", and DDSpanContext is the only ProfilerContext), so
this is defensive: route a zero span to a clean clearTraceContext instead of a
silently-swallowed throw over stale state.

Also document clearContextValue(int)'s return contract (@param/@return) and add a
testContextRegistration scenario asserting a zero-span activation does not throw and
still reapplies app context.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ical references

Rewrite or remove comments that documented the superseded DirectByteBuffer context
API and the ddprof-version history of the bridge (e.g. "Replaces the previous
setContext...", "no DBB read", the 1.41.0/1.45.0 no-op history in
DatadogProfilingScope). Those explain how the code got here, not what it does now;
the commit history and PRs carry that evolution. Comment-only, behavior unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AppContextSnapshotBenchmark.setup() still called the old
record(int, int, byte[], String) signature; AppContextSnapshot.record is now
record(int, String) (strings-only, per the all-native context migration).
compileJmhJava was failing in CI.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- setContextValue: a rejected native write (e.g. >255-byte UTF-8) clears the
  native slot but left the Java snapshot untouched, so the attribute read as
  unset until the next span boundary silently resurrected the stale prior
  value. Reapply immediately on rejection so the prior value stays visible
  continuously, matching pre-migration DBB behavior.
- setTraceContext: reapplyAppContext() ran unconditionally after the native
  call, so when profiling.context.attributes also names
  _dd.trace.operation/resource (with span-name/resource-name context
  enabled), the trailing reapply clobbered the span-derived value that
  setTraceContext just wrote to the same offset with a stale app-recorded
  one. reapplyAppContext now takes the operation/resource offsets to skip.

Both were flagged independently by Codex and Datadog Autotest PR review bots
on #11899, with a concrete repro for the first. Added regression coverage to
DatadogProfilerTest#testContextRegistration for both (verified each new
assertion fails without its corresponding fix).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@rkennke
rkennke force-pushed the rkennke/profiler-all-native-context-phase2 branch from abbe49d to 3a248ca Compare July 24, 2026 18:01
PerfectSlayer pushed a commit that referenced this pull request Jul 27, 2026
Bump java-profiler (ddprof) to 1.47.0

Required to continue work on the all-native profiler context bridge (#11899).

Merge branch 'master' into rkennke/bump-ddprof-1.47.0

Raise agent jar size budget for java-profiler 1.47.0

The ddprof 1.47.0 native library growth pushed the shadow jar past the
previous budget; raise it to unblock the verifyAgentJarContents check.

Merge branch 'master' into rkennke/bump-ddprof-1.47.0


(cherry picked from commit ac42cd4)

Co-authored-by: Roman Kennke <roman.kennke@datadoghq.com>
Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
@rkennke
rkennke requested a review from jbachorik July 27, 2026 10:46

@jbachorik jbachorik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just a small nit regarding naming.

@rkennke
rkennke added this pull request to the merge queue Jul 28, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 28, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-28 14:49:18 UTC ℹ️ Start processing command /merge


2026-07-28 14:49:23 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-07-28 16:08:05 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 28, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 8930561 into master Jul 28, 2026
605 of 607 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the rkennke/profiler-all-native-context-phase2 branch July 28, 2026 16:08
@github-actions github-actions Bot added this to the 1.65.0 milestone Jul 28, 2026
rkennke added a commit to DataDog/java-profiler that referenced this pull request Jul 29, 2026
…Writer removal

dd-trace-java's DatadogProfiler still depends on ContextSetter.offsetOf()/size()
(pure-Java attribute-slot resolution, no DBB/JNI) per its Phase 2 migration
(DataDog/dd-trace-java#11899). Deleting the class outright in Phase 3 broke that
consumer; restore just the two pure-Java methods it uses.

BufferWriter/BufferWriter8/BufferWriter9 lost their only caller (ThreadContext)
in Phase 3 but were left behind as dead code, along with the Java-9
versioned-sources/multi-release-jar wiring and sourceCompatibility javadoc
exemption that existed solely to support them. Remove all of it.
rkennke added a commit to DataDog/java-profiler that referenced this pull request Jul 31, 2026
…696)

* Remove legacy DirectByteBuffer-based OTEP context storage (Phase 3)

dd-trace-java PR #11899 has migrated onto the all-native context API
(setTraceContext/clearTraceContext/setContextValue/clearContextValue), so
java-profiler was the last consumer of the DirectByteBuffer-based
ThreadContext/OtelContextStorage path, which carried a virtual-thread
use-after-free risk. Deletes the DBB conduit and its Java/native/test/
benchmark/chaos surface, replacing coverage with equivalents against the
all-native path.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Restore minimal ContextSetter for dd-trace-java compat; finish BufferWriter removal

dd-trace-java's DatadogProfiler still depends on ContextSetter.offsetOf()/size()
(pure-Java attribute-slot resolution, no DBB/JNI) per its Phase 2 migration
(DataDog/dd-trace-java#11899). Deleting the class outright in Phase 3 broke that
consumer; restore just the two pure-Java methods it uses.

BufferWriter/BufferWriter8/BufferWriter9 lost their only caller (ThreadContext)
in Phase 3 but were left behind as dead code, along with the Java-9
versioned-sources/multi-release-jar wiring and sourceCompatibility javadoc
exemption that existed solely to support them. Remove all of it.

* Fix chaos harness default and antagonist per review feedback

vthread-context-cascade targeted a use-after-free specific to the removed
legacy DBB per-carrier-cached conduit; the all-native path resolves each
thread's context record fresh via ProfiledThread::current() on every call,
so that failure class no longer exists and there's no equivalent antagonist
to swap in. Drop the now-unknown name from run-chaos-harness.sh's
profiler+tracer default list so profiler+tracer chaos CI runs stop aborting
with "unknown antagonist".

ReapplyContextValueAntagonist.workerLoop also ignored setContextValue's
boolean return; fail fast on false so a dictionary-overflow or other
unexpected failure surfaces instead of silently losing coverage.

* Remove remaining residue from legacy DBB context removal

run-threadcontext-benchmark.sh ran a JMH class (ThreadContextBenchmark)
that no longer exists; delete it.

Fix two javadoc references to methods deleted in the same change:
clearTraceContext()'s {@link #clearContext()} and
TraceContextAntagonist's description of the tracer's activation path,
now setTraceContext/clearTraceContext.

* Make setContextValue failure in chaos antagonist actually surface

Main's own javadoc documents the harness's only failure signal as a
non-zero process exit; an exception thrown on a background pool thread
doesn't produce one on its own, and submit()'s discarded Future would
have swallowed it entirely. Switch to execute() and halt the process on
failure so an unexpected setContextValue false actually fails the job,
per the intent of the original fail-fast fix.

* Clear stale trace context before re-enabling wall-clock on reused executor threads

Removing the priming setContext(0,0) call also removed its incidental
side effect of clearing a reused worker's leftover context before
addThread() re-enables wall-clock sampling, letting a signal in the
window before ContextTask.run() activates the new context attribute
to the previous task's span.

* Address review nits: String over CharSequence, drop stale DBB comment, tidy import

setTraceContext/setContextValue take String now instead of CharSequence since
the value was immediately converted via toString() before touching the cache
either way. Also drops a comment referencing the removed DirectByteBuffer
mechanism and swaps a java.util.Arrays FQN for an import in the cache test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Clean up dead carrier-scoping build logic, default the reapply-context-value chaos antagonist, and close the custom-attribute JFR test gap

carrierExportJvmArgs()/testJvmMajorVersionFromRelease() and the two --add-exports
call sites in ProfilerTestPlugin.kt, ContextCombinedBenchmark's jvmArgsAppend, and
run-chaos-harness.sh's --add-exports line all existed solely to resolve
jdk.internal.misc.CarrierThreadLocal for OtelContextStorage's now-deleted
carrier-scoped mode; removed as dead weight.

Adds reapply-context-value to both DEFAULT_ANTAGONISTS lists so the chaos harness
exercises the antagonist that specifically targets the UAF class this PR removes,
by default rather than only via ANTAGONISTS_OVERRIDE.

Adds CustomContextAttributeSamplingTest, porting the weight-distribution/
jdk.ActiveSetting/dictionary_context_keys assertions from the deleted
TagContextTest onto the all-native setContextValue API, closing the coverage gap
between a value written through that API and it surfacing as a correctly-named,
correctly-weighted JFR field.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Keep setTraceContext's activation attributes as CharSequence, and make the cache honor it

dd-trace-java's DatadogProfiler.setTraceContext forwards
ProfilerContext#getOperationName()/getResourceName() directly into this method,
both declared CharSequence throughout DDSpanContext's public span API —
UTF8BytesString and SubSequence exist specifically to avoid materializing a
String for these fields on the tracing hot path. Narrowing to String here would
break that call site at compile time on the next ddprof-lib version bump, and
runs counter to the tracer's own allocation-avoidance idiom. setContextValue
keeps its String param, since dd-trace-java always passes an actual String
there.

Since the whole point of accepting CharSequence is avoiding an eager String,
ContextValueCache gains a resolve(CharSequence) overload that never calls
toString() on a cache hit: contentHashCode() replicates String#hashCode()'s
documented polynomial algorithm (part of the API contract, not an
implementation detail) so a CharSequence and a content-equal cached String key
land in the same slot, and String#contentEquals(CharSequence) compares without
allocating. toString() is only paid on a genuine miss, where registerConstant0
needs a real String anyway.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Remove ambiguous String/CharSequence overload pair on ContextValueCache.resolve

resolve(String) and resolve(CharSequence) as separate overloads meant dispatch
depended on the caller's static type rather than the value's runtime type
(e.g. JavaProfiler.resolveContextValue's CharSequence-typed local always binds
to the CharSequence overload even when it happens to hold a String) — correct
today only because of the explicit instanceof check, but a footgun for future
callers who might assume Java picks the more specific runtime match. resolve
is now the single entrypoint; the String-only fast path moves to a private
resolveString helper.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
@PerfectSlayer PerfectSlayer changed the title profiling(ddprof): migrate context bridge to the all-native API (Phase 2) Migrate context bridge to the all-native DDProf API (Phase 2) Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: profiling Profiling tag: ai generated Largely based on code generated by an AI or LLM type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants