Migrate context bridge to the all-native DDProf API (Phase 2) - #11899
Conversation
Bump java-profiler (ddprof) to 1.47.0 Required to continue work on the all-native profiler context bridge (#11899). Merge branch 'master' into rkennke/bump-ddprof-1.47.0 Raise agent jar size budget for java-profiler 1.47.0 The ddprof 1.47.0 native library growth pushed the shadow jar past the previous budget; raise it to unblock the verifyAgentJarContents check. Merge branch 'master' into rkennke/bump-ddprof-1.47.0 Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
b1d70ef to
07a20b2
Compare
|
🎯 Code Coverage (details) 🔗 Commit SHA: 3668c1f | Docs | Datadog PR Page | Give us feedback! |
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 07a20b2f98
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Replacing an existing profiling context value with a >255-byte string now clears the native slot but retains its Java snapshot; the next span activation resurrects the stale prior value. The DBB base path retains the prior value continuously, so samples can lose and later regain incorrect context across this migration.
📊 Validated against 10 scenarios · Open Bits AI session
🤖 Datadog Autotest · Commit 07a20b2 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
- setContextValue: a rejected native write (e.g. >255-byte UTF-8) clears the native slot but left the Java snapshot untouched, so the attribute read as unset until the next span boundary silently resurrected the stale prior value. Reapply immediately on rejection so the prior value stays visible continuously, matching pre-migration DBB behavior. - setTraceContext: reapplyAppContext() ran unconditionally after the native call, so when profiling.context.attributes also names _dd.trace.operation/resource (with span-name/resource-name context enabled), the trailing reapply clobbered the span-derived value that setTraceContext just wrote to the same offset with a stale app-recorded one. reapplyAppContext now takes the operation/resource offsets to skip. Both were flagged independently by Codex and Datadog Autotest PR review bots on #11899, with a concrete repro for the first. Added regression coverage to DatadogProfilerTest#testContextRegistration for both (verified each new assertion fails without its corresponding fix). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Switch the profiler context bridge off the deprecated DirectByteBuffer (DBB) context API onto java-profiler's all-native API (setTraceContext / clearTraceContext / setContextValue / clearContextValue). This eliminates the virtual-thread use-after-free (the DBB cached buffer that dangled on carrier migration) and folds the per-activation sequence (setContext + two setContextValue) into a single native call. - DatadogProfilingIntegration.activate: one setTraceContext(...) carrying trace/span context + operation and resource attributes (3 JNI calls -> 1); close/clearContext: clearTraceContext() (wipes op/resource slots too). - DatadogProfiler: setContextValue/clearContextValue/reapplyAppContext/ syncNativeAppContext are now all-native. reapplyAppContext uses a per-slot native setContextValue loop (native setContextValue publishes valid=1, so app context stays visible without an active span — preserves PR #11646). A native batch reapply is deferred to a measured follow-up (java-profiler PROF-15361). - AppContextSnapshot simplified to strings-only (the native path resolves each value's encoding via the process-wide cache; no cached id/utf8/snapshotTags). - snapshot() uses the new native copyContextTags read (no ThreadContext/DBB, so it observes native writes without resetting the record). - ContextSetter kept only for offsetOf + size (pure Java); no DBB usage remains. Requires java-profiler with the all-native API (ddprof >= the phase-1 release; DataDog/java-profiler#631). Build/test with -PddprofUseSnapshot=true against a local publishToMavenLocal 1.47.0-SNAPSHOT until that ships. ddprof suite is Linux-gated (assumeTrue(isLinux)) — verify on Linux/CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ContextValue The native setTraceContext rejects spanId==0 with IllegalArgumentException (it is the activation path; clearing is clearTraceContext). The bridge's catch(Throwable) would swallow that throw and leave the previous span's context stale on the thread. Span ids are non-zero by construction (IdGenerationStrategy never yields 0, DDSpanId.ZERO means "no span", and DDSpanContext is the only ProfilerContext), so this is defensive: route a zero span to a clean clearTraceContext instead of a silently-swallowed throw over stale state. Also document clearContextValue(int)'s return contract (@param/@return) and add a testContextRegistration scenario asserting a zero-span activation does not throw and still reapplies app context. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ical references Rewrite or remove comments that documented the superseded DirectByteBuffer context API and the ddprof-version history of the bridge (e.g. "Replaces the previous setContext...", "no DBB read", the 1.41.0/1.45.0 no-op history in DatadogProfilingScope). Those explain how the code got here, not what it does now; the commit history and PRs carry that evolution. Comment-only, behavior unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AppContextSnapshotBenchmark.setup() still called the old record(int, int, byte[], String) signature; AppContextSnapshot.record is now record(int, String) (strings-only, per the all-native context migration). compileJmhJava was failing in CI. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- setContextValue: a rejected native write (e.g. >255-byte UTF-8) clears the native slot but left the Java snapshot untouched, so the attribute read as unset until the next span boundary silently resurrected the stale prior value. Reapply immediately on rejection so the prior value stays visible continuously, matching pre-migration DBB behavior. - setTraceContext: reapplyAppContext() ran unconditionally after the native call, so when profiling.context.attributes also names _dd.trace.operation/resource (with span-name/resource-name context enabled), the trailing reapply clobbered the span-derived value that setTraceContext just wrote to the same offset with a stale app-recorded one. reapplyAppContext now takes the operation/resource offsets to skip. Both were flagged independently by Codex and Datadog Autotest PR review bots on #11899, with a concrete repro for the first. Added regression coverage to DatadogProfilerTest#testContextRegistration for both (verified each new assertion fails without its corresponding fix). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
abbe49d to
3a248ca
Compare
Bump java-profiler (ddprof) to 1.47.0 Required to continue work on the all-native profiler context bridge (#11899). Merge branch 'master' into rkennke/bump-ddprof-1.47.0 Raise agent jar size budget for java-profiler 1.47.0 The ddprof 1.47.0 native library growth pushed the shadow jar past the previous budget; raise it to unblock the verifyAgentJarContents check. Merge branch 'master' into rkennke/bump-ddprof-1.47.0 (cherry picked from commit ac42cd4) Co-authored-by: Roman Kennke <roman.kennke@datadoghq.com> Co-authored-by: devflow.devflow-routing-intake <devflow.devflow-routing-intake@kubernetes.us1.ddbuild.io>
jbachorik
left a comment
There was a problem hiding this comment.
Looks good. Just a small nit regarding naming.
…ceOffset Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
8930561
into
master
…Writer removal dd-trace-java's DatadogProfiler still depends on ContextSetter.offsetOf()/size() (pure-Java attribute-slot resolution, no DBB/JNI) per its Phase 2 migration (DataDog/dd-trace-java#11899). Deleting the class outright in Phase 3 broke that consumer; restore just the two pure-Java methods it uses. BufferWriter/BufferWriter8/BufferWriter9 lost their only caller (ThreadContext) in Phase 3 but were left behind as dead code, along with the Java-9 versioned-sources/multi-release-jar wiring and sourceCompatibility javadoc exemption that existed solely to support them. Remove all of it.
…696) * Remove legacy DirectByteBuffer-based OTEP context storage (Phase 3) dd-trace-java PR #11899 has migrated onto the all-native context API (setTraceContext/clearTraceContext/setContextValue/clearContextValue), so java-profiler was the last consumer of the DirectByteBuffer-based ThreadContext/OtelContextStorage path, which carried a virtual-thread use-after-free risk. Deletes the DBB conduit and its Java/native/test/ benchmark/chaos surface, replacing coverage with equivalents against the all-native path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Restore minimal ContextSetter for dd-trace-java compat; finish BufferWriter removal dd-trace-java's DatadogProfiler still depends on ContextSetter.offsetOf()/size() (pure-Java attribute-slot resolution, no DBB/JNI) per its Phase 2 migration (DataDog/dd-trace-java#11899). Deleting the class outright in Phase 3 broke that consumer; restore just the two pure-Java methods it uses. BufferWriter/BufferWriter8/BufferWriter9 lost their only caller (ThreadContext) in Phase 3 but were left behind as dead code, along with the Java-9 versioned-sources/multi-release-jar wiring and sourceCompatibility javadoc exemption that existed solely to support them. Remove all of it. * Fix chaos harness default and antagonist per review feedback vthread-context-cascade targeted a use-after-free specific to the removed legacy DBB per-carrier-cached conduit; the all-native path resolves each thread's context record fresh via ProfiledThread::current() on every call, so that failure class no longer exists and there's no equivalent antagonist to swap in. Drop the now-unknown name from run-chaos-harness.sh's profiler+tracer default list so profiler+tracer chaos CI runs stop aborting with "unknown antagonist". ReapplyContextValueAntagonist.workerLoop also ignored setContextValue's boolean return; fail fast on false so a dictionary-overflow or other unexpected failure surfaces instead of silently losing coverage. * Remove remaining residue from legacy DBB context removal run-threadcontext-benchmark.sh ran a JMH class (ThreadContextBenchmark) that no longer exists; delete it. Fix two javadoc references to methods deleted in the same change: clearTraceContext()'s {@link #clearContext()} and TraceContextAntagonist's description of the tracer's activation path, now setTraceContext/clearTraceContext. * Make setContextValue failure in chaos antagonist actually surface Main's own javadoc documents the harness's only failure signal as a non-zero process exit; an exception thrown on a background pool thread doesn't produce one on its own, and submit()'s discarded Future would have swallowed it entirely. Switch to execute() and halt the process on failure so an unexpected setContextValue false actually fails the job, per the intent of the original fail-fast fix. * Clear stale trace context before re-enabling wall-clock on reused executor threads Removing the priming setContext(0,0) call also removed its incidental side effect of clearing a reused worker's leftover context before addThread() re-enables wall-clock sampling, letting a signal in the window before ContextTask.run() activates the new context attribute to the previous task's span. * Address review nits: String over CharSequence, drop stale DBB comment, tidy import setTraceContext/setContextValue take String now instead of CharSequence since the value was immediately converted via toString() before touching the cache either way. Also drops a comment referencing the removed DirectByteBuffer mechanism and swaps a java.util.Arrays FQN for an import in the cache test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Clean up dead carrier-scoping build logic, default the reapply-context-value chaos antagonist, and close the custom-attribute JFR test gap carrierExportJvmArgs()/testJvmMajorVersionFromRelease() and the two --add-exports call sites in ProfilerTestPlugin.kt, ContextCombinedBenchmark's jvmArgsAppend, and run-chaos-harness.sh's --add-exports line all existed solely to resolve jdk.internal.misc.CarrierThreadLocal for OtelContextStorage's now-deleted carrier-scoped mode; removed as dead weight. Adds reapply-context-value to both DEFAULT_ANTAGONISTS lists so the chaos harness exercises the antagonist that specifically targets the UAF class this PR removes, by default rather than only via ANTAGONISTS_OVERRIDE. Adds CustomContextAttributeSamplingTest, porting the weight-distribution/ jdk.ActiveSetting/dictionary_context_keys assertions from the deleted TagContextTest onto the all-native setContextValue API, closing the coverage gap between a value written through that API and it surfacing as a correctly-named, correctly-weighted JFR field. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Keep setTraceContext's activation attributes as CharSequence, and make the cache honor it dd-trace-java's DatadogProfiler.setTraceContext forwards ProfilerContext#getOperationName()/getResourceName() directly into this method, both declared CharSequence throughout DDSpanContext's public span API — UTF8BytesString and SubSequence exist specifically to avoid materializing a String for these fields on the tracing hot path. Narrowing to String here would break that call site at compile time on the next ddprof-lib version bump, and runs counter to the tracer's own allocation-avoidance idiom. setContextValue keeps its String param, since dd-trace-java always passes an actual String there. Since the whole point of accepting CharSequence is avoiding an eager String, ContextValueCache gains a resolve(CharSequence) overload that never calls toString() on a cache hit: contentHashCode() replicates String#hashCode()'s documented polynomial algorithm (part of the API contract, not an implementation detail) so a CharSequence and a content-equal cached String key land in the same slot, and String#contentEquals(CharSequence) compares without allocating. toString() is only paid on a genuine miss, where registerConstant0 needs a real String anyway. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Remove ambiguous String/CharSequence overload pair on ContextValueCache.resolve resolve(String) and resolve(CharSequence) as separate overloads meant dispatch depended on the caller's static type rather than the value's runtime type (e.g. JavaProfiler.resolveContextValue's CharSequence-typed local always binds to the CharSequence overload even when it happens to hold a String) — correct today only because of the explicit instanceof check, but a footgun for future callers who might assume Java picks the more specific runtime match. resolve is now the single entrypoint; the String-only fast path moves to a private resolveString helper. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
What
Migrates the profiler context bridge off the deprecated DirectByteBuffer (DBB) context API onto java-profiler's all-native API (
setTraceContext/clearTraceContext/setContextValue/clearContextValue).This is Phase 2 of the all-native context work (Phase 1 = DataDog/java-profiler#631).
Why
DirectByteBufferover the native OTEP record; on a carrier that a mounted virtual thread later migrated off, the record was freed while the cached buffer kept being written. The all-native API resolves the current carrier's record inside each JNI call — no cached buffer to dangle.setContext+ 2×setContextValue(3 JNI calls) into onesetTraceContext(~28% faster on the combined cycle per the design-note benchmark).Changes
DatadogProfilingIntegration.activate→ onesetTraceContext(...)carrying trace/span context + operation & resource attributes;close/clearContext→clearTraceContext().DatadogProfiler:setContextValue/clearContextValue/reapplyAppContext/syncNativeAppContextare now all-native.reapplyAppContextuses a per-slot nativesetContextValueloop — nativesetContextValuepublishes the record (valid=1) even with no active span, so app context stays visible between spans (preserves Restore app context attributes wiped by ddprof setContext on span activation #11646). A native batch reapply is deferred to a measured follow-up (java-profiler PROF-15361).AppContextSnapshotsimplified to strings-only (the native path resolves each value's encoding via the process-wide cache — no cached id/utf8/snapshotTagsDBB read).snapshot()uses the new nativecopyContextTagsread (noThreadContext/DBB, so it observes native writes without resetting the record).ContextSetterkept only foroffsetOf+size(pure Java); no DBB usage remains.Behavior notes
reapplyAppContextpublishes each app-managed attribute with its own nativesetContextValue(detach→attach), so a profiler sample landing mid-reapply can observe a partial app-attribute set on activation — a window the old batch fast-path (setContextValuesByIdAndBytes, single attach) did not have. This is deliberate for the expected small app-attribute cardinality (0–2); the single-call batch reapply is the measured follow-up (PROF-15361). Trace/span + operation/resource stay atomic (onesetTraceContext).setTraceContextrejectsspanId==0(it is the activation path; clearing isclearTraceContext). Span ids are non-zero by construction (IdGenerationStrategynever yields 0,DDSpanId.ZEROmeans "no span", andDDSpanContextis the onlyProfilerContext), so the bridge routes a stray zero span toclearTraceContext()rather than letting theIllegalArgumentExceptionbe swallowed over stale context.Dependency
Requires java-profiler's all-native API, from DataDog/java-profiler#631. Master now pins
ddprof = 1.47.0(#12055), which includes it, so this branch has been rebased and builds against the published release — no local snapshot needed.Status / verification
ddprof:1.47.0(Bump java-profiler (ddprof) to 1.47.0 #12055); compiles clean (nocannot find symbolon the new calls) and theprofiling-ddprofunit test suite is GREEN on Linux post-rebase.ddprof:1.47.0release (native lib extracted from the published jar, not a local snapshot):dd-smoke-tests:profiling-integration-tests(CodeHotspotsTest,JFRBasedProfilingIntegrationTest) — 20 tests, 2 skipped (both pre-existing/expected), 0 failures, 0 errors (end-to-end agent + profiler + JFR, exercising span/op/resource context).setContextValue, andreapplyAppContextoverwriting freshly-written operation/resource values) were fixed and covered by new regression assertions inDatadogProfilerTest, each confirmed to fail without its corresponding fix.🤖 Generated with Claude Code