Skip to content

[crashtracker/profiler] Notify profiler the application is crashing#7657

Merged
gleocadie merged 1 commit intomasterfrom
gleocadie/fix-profiler-crashing-an-already-crashing-app
Oct 17, 2025
Merged

[crashtracker/profiler] Notify profiler the application is crashing#7657
gleocadie merged 1 commit intomasterfrom
gleocadie/fix-profiler-crashing-an-already-crashing-app

Conversation

@gleocadie
Copy link
Collaborator

@gleocadie gleocadie commented Oct 15, 2025 •
edited
Loading

Summary of changes

Nth attempt to prevent the profiler from collecting callstacks when the application is crashing.

Reason for change

We have this issue for a while and this appeared in some crash reports lately. The profiler is trying to collect callstacks while the app is crashing. This can lead to a crash on a crash and appearance of zombie processes.

syscall (sysdeps/unix/sysv/linux/x86_64/syscall.S:38)
_write_validate (/project/obj/libunwind-prefix/src/libunwind/src/mi/Gaddress_validator.c:140)
access_mem (/project/obj/libunwind-prefix/src/libunwind/src/x86_64/Ginit.c:89)
dwarf_get (/project/obj/libunwind-prefix/src/libunwind/include/tdep-x86_64/libunwind_i.h:205)
unw_backtrace2 (/project/obj/libunwind-prefix/src/libunwind/src/mi/backtrace.c:111)
LinuxStackFramesCollector::CollectStackWithBacktrace2(void*) (/project/profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/LinuxStackFramesCollector.cpp:285)
LinuxStackFramesCollector::CollectStackSampleSignalHandler(int, siginfo_t*, void*) (/project/profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/LinuxStackFramesCollector.cpp)
ProfilerSignalManager::SignalHandler(int, siginfo_t*, void*) (/project/profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/ProfilerSignalManager.cpp:183)
__restore_rt
__GI___wait4 (sysdeps/unix/sysv/linux/wait4.c:30)
PROCCreateCrashDump(std::vector<char const*, std::allocator<char const*> >&, char*, int, bool) (/__w/1/s/src/coreclr/pal/src/thread/process.cpp:2545)
PROCCreateCrashDumpIfEnabled (/__w/1/s/src/coreclr/pal/src/thread/process.cpp)
PROCAbort (/__w/1/s/src/coreclr/pal/src/thread/process.cpp:2793)
PROCEndProcess(void*, unsigned int, int) (/__w/1/s/src/coreclr/pal/src/thread/process.cpp:1355)
UnwindManagedExceptionPass1(PAL_SEHException&, _CONTEXT*) (/__w/1/s/src/coreclr/vm/exceptionhandling.cpp)
DispatchManagedException(PAL_SEHException&, bool) (/__w/1/s/src/coreclr/vm/exceptionhandling.cpp)
IL_Throw(Object*) (/__w/1/s/src/coreclr/vm/jithelpers.cpp)

When the application is crashing, the .NET runtime calls fork(). Calling this function leads to the creation of a new process (child process). This process will have a copy of the parent address space.
Then the .NET runtime, in the child process, calls execve() to run createdump. We wrapped execve() in order to call our own tool which will create a crash report and send it to our backend.
In our implementation of execve(), we set a flag to notify the profiler to not collect callstack because the application is crashing.

The current implementation does not work because we use an atomic_int which will be copied and when the child process set flag, it will be its own version of the flag. The parent process won't be able to see it changed. We need a shared memory so the parent and child processes can communicate.

Implementation details

  • create a shared memory region (using mmap)

Test coverage

We should not see issue caused by the profiler in our CI when crashes happen

Other details

@kevingosse , I finally found it :)

@github-actions github-actions bot added the area:profiler Issues related to the continous-profiler label Oct 15, 2025
andrewlock
andrewlock approved these changes Oct 15, 2025
View reviewed changes
Copy link
Member

@andrewlock andrewlock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooooh, nice one!

@dd-trace-dotnet-ci-bot
Copy link

dd-trace-dotnet-ci-bot bot commented Oct 15, 2025 •
edited
Loading

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (71ms)  : 70, 72
     .   : milestone, 71,
    master - mean (71ms)  : 71, 72
     .   : milestone, 71,

    section Baseline
    This PR (7657) - mean (68ms)  : 66, 70
     .   : milestone, 68,
    master - mean (68ms)  : 66, 70
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (1,055ms)  : 984, 1127
     .   : milestone, 1055,
    master - mean (1,048ms)  : 1003, 1093
     .   : milestone, 1048,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (106ms)  : 105, 107
     .   : milestone, 106,
    master - mean (106ms)  : 104, 107
     .   : milestone, 106,

    section Baseline
    This PR (7657) - mean (105ms)  : 103, 107
     .   : milestone, 105,
    master - mean (105ms)  : 102, 108
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (743ms)  : 717, 768
     .   : milestone, 743,
    master - mean (744ms)  : 713, 774
     .   : milestone, 744,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (100ms)  : 99, 101
     .   : milestone, 100,
    master - mean (101ms)  : 100, 102
     .   : milestone, 101,

    section Baseline
    This PR (7657) - mean (100ms)  : 98, 102
     .   : milestone, 100,
    master - mean (99ms)  : 97, 101
     .   : milestone, 99,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (769ms)  : 722, 816
     .   : milestone, 769,
    master - mean (781ms)  : 747, 814
     .   : milestone, 781,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (93ms)  : 91, 94
     .   : milestone, 93,
    master - mean (92ms)  : 91, 94
     .   : milestone, 92,

    section Baseline
    This PR (7657) - mean (92ms)  : 89, 95
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (657ms)  : 646, 669
     .   : milestone, 657,
    master - mean (660ms)  : 650, 670
     .   : milestone, 660,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (196ms)  : 193, 198
     .   : milestone, 196,
    master - mean (195ms)  : 193, 198
     .   : milestone, 195,

    section Baseline
    This PR (7657) - mean (192ms)  : 189, 194
     .   : milestone, 192,
    master - mean (193ms)  : 188, 197
     .   : milestone, 193,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (1,162ms)  : 1108, 1216
     .   : milestone, 1162,
    master - mean (1,172ms)  : 1106, 1239
     .   : milestone, 1172,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (278ms)  : 272, 283
     .   : milestone, 278,
    master - mean (277ms)  : 273, 280
     .   : milestone, 277,

    section Baseline
    This PR (7657) - mean (277ms)  : 272, 281
     .   : milestone, 277,
    master - mean (276ms)  : 271, 280
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (944ms)  : 889, 999
     .   : milestone, 944,
    master - mean (946ms)  : 906, 986
     .   : milestone, 946,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (281ms)  : 276, 286
     .   : milestone, 281,
    master - mean (279ms)  : 275, 283
     .   : milestone, 279,

    section Baseline
    This PR (7657) - mean (280ms)  : 276, 285
     .   : milestone, 280,
    master - mean (279ms)  : 274, 284
     .   : milestone, 279,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (992ms)  : 958, 1026
     .   : milestone, 992,
    master - mean (997ms)  : 965, 1030
     .   : milestone, 997,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7657) - mean (269ms)  : 265, 273
     .   : milestone, 269,
    master - mean (269ms)  : 265, 274
     .   : milestone, 269,

    section Baseline
    This PR (7657) - mean (268ms)  : 264, 272
     .   : milestone, 268,
    master - mean (269ms)  : 265, 273
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (7657) - mean (847ms)  : 834, 860
     .   : milestone, 847,
    master - mean (854ms)  : 837, 871
     .   : milestone, 854,
Loading

@gleocadie gleocadie force-pushed the gleocadie/fix-profiler-crashing-an-already-crashing-app branch from 5177554 to b08bd88 Compare October 16, 2025 08:23
@gleocadie gleocadie force-pushed the gleocadie/fix-profiler-crashing-an-already-crashing-app branch from b08bd88 to b033321 Compare October 16, 2025 08:29
@gleocadie gleocadie marked this pull request as ready for review October 16, 2025 08:46
@gleocadie gleocadie requested a review from a team as a code owner October 16, 2025 08:46
@pr-commenter
Copy link

pr-commenter bot commented Oct 16, 2025 •
edited
Loading

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7657 compared to master:

  • 1 benchmarks are faster, with geometric mean 2.413
  • 8 benchmarks have fewer allocations
  • 6 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7657

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑netcoreapp3.1 5.66 KB 5.73 KB 64 B 1.13%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net472 6.08 KB 6.12 KB 35 B 0.58%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 10.4μs 56.4ns 334ns 0 0 0 5.52 KB
master StartStopWithChild netcoreapp3.1 13.2μs 67.5ns 309ns 0 0 0 5.66 KB
master StartStopWithChild net472 22.4μs 122ns 710ns 0.927 0.206 0 6.08 KB
#7657 StartStopWithChild net6.0 10.8μs 60.4ns 387ns 0 0 0 5.5 KB
#7657 StartStopWithChild netcoreapp3.1 13.9μs 72.2ns 338ns 0 0 0 5.73 KB
#7657 StartStopWithChild net472 22.1μs 114ns 560ns 0.907 0.34 0 6.12 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 956μs 197ns 738ns 0 0 0 2.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 1.02ms 201ns 777ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 1.21ms 96.9ns 375ns 0 0 0 3.31 KB
#7657 WriteAndFlushEnrichedTraces net6.0 936μs 317ns 1.23μs 0 0 0 2.71 KB
#7657 WriteAndFlushEnrichedTraces netcoreapp3.1 1.01ms 286ns 1.07μs 0 0 0 2.7 KB
#7657 WriteAndFlushEnrichedTraces net472 1.19ms 675ns 2.34μs 0 0 0 3.31 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7657

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑netcoreapp3.1 189.23 KB 187.44 KB -1.79 KB -0.95%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0 183.55 KB 181.79 KB -1.76 KB -0.96%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑netcoreapp3.1 185.8 KB 184.01 KB -1.79 KB -0.96%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0 180.01 KB 178.25 KB -1.76 KB -0.98%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472 209.83 KB 207.5 KB -2.33 KB -1.11%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 206.3 KB 203.96 KB -2.34 KB -1.14%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 361μs 201ns 696ns 0 0 0 180.01 KB
master AllCycleSimpleBody netcoreapp3.1 495μs 2.38μs 9.52μs 0 0 0 185.8 KB
master AllCycleSimpleBody net472 471μs 74.2ns 257ns 32.4 0 0 206.3 KB
master AllCycleMoreComplexBody net6.0 361μs 222ns 769ns 0 0 0 183.55 KB
master AllCycleMoreComplexBody netcoreapp3.1 555μs 1.38μs 5.35μs 0 0 0 189.23 KB
master AllCycleMoreComplexBody net472 482μs 100ns 376ns 31.2 0 0 209.83 KB
master ObjectExtractorSimpleBody net6.0 315ns 1.7ns 8.5ns 0 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 406ns 2.07ns 9.69ns 0 0 0 272 B
master ObjectExtractorSimpleBody net472 300ns 0.0302ns 0.117ns 0.0438 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 6.24μs 30.1ns 121ns 0 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 7.87μs 36.5ns 146ns 0 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 6.64μs 0.835ns 3.23ns 0.598 0 0 3.8 KB
#7657 AllCycleSimpleBody net6.0 345μs 1.41μs 5.46μs 0 0 0 178.25 KB
#7657 AllCycleSimpleBody netcoreapp3.1 520μs 387ns 1.5μs 0 0 0 184.01 KB
#7657 AllCycleSimpleBody net472 460μs 121ns 469ns 31.2 0 0 203.96 KB
#7657 AllCycleMoreComplexBody net6.0 348μs 1.34μs 5.01μs 0 0 0 181.79 KB
#7657 AllCycleMoreComplexBody netcoreapp3.1 537μs 1.09μs 3.94μs 0 0 0 187.44 KB
#7657 AllCycleMoreComplexBody net472 469μs 69ns 249ns 32.4 0 0 207.5 KB
#7657 ObjectExtractorSimpleBody net6.0 318ns 1.78ns 9.6ns 0 0 0 280 B
#7657 ObjectExtractorSimpleBody netcoreapp3.1 386ns 2.19ns 15.5ns 0 0 0 272 B
#7657 ObjectExtractorSimpleBody net472 304ns 0.00821ns 0.0307ns 0.0444 0 0 281 B
#7657 ObjectExtractorMoreComplexBody net6.0 6.42μs 28.4ns 98.3ns 0 0 0 3.78 KB
#7657 ObjectExtractorMoreComplexBody netcoreapp3.1 7.75μs 1.44ns 5.2ns 0 0 0 3.69 KB
#7657 ObjectExtractorMoreComplexBody net472 6.68μs 2.86ns 11.1ns 0.57 0 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 76.8μs 255ns 954ns 0 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 95.9μs 272ns 1.05μs 0 0 0 32.4 KB
master EncodeArgs net472 111μs 14.9ns 57.7ns 4.99 0 0 32.51 KB
master EncodeLegacyArgs net6.0 144μs 50.3ns 174ns 0 0 0 2.15 KB
master EncodeLegacyArgs netcoreapp3.1 196μs 205ns 792ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 261μs 23.6ns 88.4ns 0 0 0 2.16 KB
#7657 EncodeArgs net6.0 77.5μs 43ns 161ns 0 0 0 32.4 KB
#7657 EncodeArgs netcoreapp3.1 96.5μs 269ns 1.04μs 0 0 0 32.4 KB
#7657 EncodeArgs net472 110μs 69.4ns 269ns 4.95 0 0 32.51 KB
#7657 EncodeLegacyArgs net6.0 145μs 12.2ns 47.2ns 0 0 0 2.14 KB
#7657 EncodeLegacyArgs netcoreapp3.1 197μs 189ns 731ns 0 0 0 2.14 KB
#7657 EncodeLegacyArgs net472 262μs 28ns 108ns 0 0 0 2.16 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #7657

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑netcoreapp3.1 2.413 717,336.31 297,340.92

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 395μs 99ns 370ns 0 0 0 4.55 KB
master RunWafRealisticBenchmark netcoreapp3.1 807μs 14μs 140μs 0 0 0 4.48 KB
master RunWafRealisticBenchmark net472 434μs 43.1ns 167ns 0 0 0 4.66 KB
master RunWafRealisticBenchmarkWithAttack net6.0 284μs 25.7ns 96.3ns 0 0 0 2.24 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 673μs 12.9μs 128μs 0 0 0 2.22 KB
master RunWafRealisticBenchmarkWithAttack net472 311μs 23ns 89.2ns 0 0 0 2.29 KB
#7657 RunWafRealisticBenchmark net6.0 396μs 111ns 385ns 0 0 0 4.55 KB
#7657 RunWafRealisticBenchmark netcoreapp3.1 801μs 10.7μs 106μs 0 0 0 4.48 KB
#7657 RunWafRealisticBenchmark net472 435μs 45.7ns 165ns 0 0 0 4.66 KB
#7657 RunWafRealisticBenchmarkWithAttack net6.0 286μs 85.8ns 332ns 0 0 0 2.24 KB
#7657 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 297μs 42.3ns 153ns 0 0 0 2.22 KB
#7657 RunWafRealisticBenchmarkWithAttack net472 310μs 27.4ns 106ns 0 0 0 2.29 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 59.9μs 62.5ns 242ns 0 0 0 14.52 KB
master SendRequest netcoreapp3.1 71.8μs 123ns 460ns 0 0 0 17.42 KB
master SendRequest net472 0.00439ns 0.0016ns 0.00618ns 0 0 0 0 b
#7657 SendRequest net6.0 60.9μs 93.5ns 324ns 0 0 0 14.52 KB
#7657 SendRequest netcoreapp3.1 72.7μs 83.5ns 301ns 0 0 0 17.42 KB
#7657 SendRequest net472 0.00537ns 0.00225ns 0.0087ns 0 0 0 0 b
Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7657

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0 3 B 4 B 1 B 33.33%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master OriginalCharSlice net6.0 2.02ms 4.33μs 16.8μs 0 0 0 640 KB
master OriginalCharSlice netcoreapp3.1 2.08ms 5.68μs 22μs 0 0 0 640 KB
master OriginalCharSlice net472 2.7ms 413ns 1.6μs 100 0 0 641.95 KB
master OptimizedCharSlice net6.0 1.38ms 121ns 469ns 0 0 0 4 B
master OptimizedCharSlice netcoreapp3.1 1.79ms 280ns 1.05μs 0 0 0 1 B
master OptimizedCharSlice net472 1.94ms 212ns 820ns 0 0 0 73 B
master OptimizedCharSliceWithPool net6.0 890μs 51.2ns 198ns 0 0 0 3 B
master OptimizedCharSliceWithPool netcoreapp3.1 798μs 81.2ns 314ns 0 0 0 0 b
master OptimizedCharSliceWithPool net472 1.15ms 76ns 284ns 0 0 0 47 B
#7657 OriginalCharSlice net6.0 1.94ms 3.79μs 14.7μs 0 0 0 640 KB
#7657 OriginalCharSlice netcoreapp3.1 2.12ms 2.97μs 10.7μs 0 0 0 640 KB
#7657 OriginalCharSlice net472 2.65ms 699ns 2.71μs 100 0 0 641.95 KB
#7657 OptimizedCharSlice net6.0 1.38ms 185ns 715ns 0 0 0 4 B
#7657 OptimizedCharSlice netcoreapp3.1 1.74ms 230ns 892ns 0 0 0 1 B
#7657 OptimizedCharSlice net472 1.95ms 369ns 1.43μs 0 0 0 73 B
#7657 OptimizedCharSliceWithPool net6.0 852μs 54ns 209ns 0 0 0 4 B
#7657 OptimizedCharSliceWithPool netcoreapp3.1 835μs 122ns 441ns 0 0 0 0 b
#7657 OptimizedCharSliceWithPool net472 1.14ms 93.7ns 350ns 0 0 0 47 B
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7657

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.63 KB 42.32 KB 692 B 1.66%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 719μs 3.88μs 21.2μs 0 0 0 41.63 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 665μs 3.11μs 12μs 0 0 0 42.03 KB
master WriteAndFlushEnrichedTraces net472 920μs 3.34μs 12.9μs 8.33 0 0 56.08 KB
#7657 WriteAndFlushEnrichedTraces net6.0 635μs 542ns 2.1μs 0 0 0 42.32 KB
#7657 WriteAndFlushEnrichedTraces netcoreapp3.1 692μs 679ns 2.35μs 0 0 0 42.02 KB
#7657 WriteAndFlushEnrichedTraces net472 940μs 4.55μs 19.3μs 4.46 0 0 55.97 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.9μs 9.15ns 35.4ns 0 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 2.56μs 7.07ns 24.5ns 0 0 0 1.02 KB
master ExecuteNonQuery net472 2.91μs 4.67ns 17.5ns 0.148 0.0148 0 987 B
#7657 ExecuteNonQuery net6.0 1.93μs 1.78ns 6.89ns 0 0 0 1.02 KB
#7657 ExecuteNonQuery netcoreapp3.1 2.59μs 6.85ns 26.5ns 0 0 0 1.02 KB
#7657 ExecuteNonQuery net472 2.78μs 4.75ns 18.4ns 0.152 0.0138 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.77μs 8.57ns 32.1ns 0 0 0 1.03 KB
master CallElasticsearch netcoreapp3.1 2.22μs 10.6ns 42.5ns 0 0 0 1.03 KB
master CallElasticsearch net472 3.59μs 1.37ns 4.74ns 0.162 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.85μs 6.67ns 25ns 0 0 0 1.01 KB
master CallElasticsearchAsync netcoreapp3.1 2.42μs 3.67ns 14.2ns 0 0 0 1.08 KB
master CallElasticsearchAsync net472 3.7μs 3.61ns 13.5ns 0.167 0 0 1.1 KB
#7657 CallElasticsearch net6.0 1.79μs 2.56ns 9.9ns 0 0 0 1.03 KB
#7657 CallElasticsearch netcoreapp3.1 2.33μs 7.65ns 27.6ns 0 0 0 1.03 KB
#7657 CallElasticsearch net472 3.44μs 1.63ns 5.65ns 0.155 0 0 1.04 KB
#7657 CallElasticsearchAsync net6.0 1.91μs 4.84ns 18.8ns 0 0 0 1.01 KB
#7657 CallElasticsearchAsync netcoreapp3.1 2.46μs 10.3ns 37.1ns 0 0 0 1.08 KB
#7657 CallElasticsearchAsync net472 3.68μs 1.43ns 5.15ns 0.166 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.78μs 1.9ns 6.86ns 0 0 0 952 B
master ExecuteAsync netcoreapp3.1 2.31μs 6.04ns 23.4ns 0 0 0 952 B
master ExecuteAsync net472 2.47μs 2.27ns 8.79ns 0.137 0 0 915 B
#7657 ExecuteAsync net6.0 1.88μs 3.79ns 14.7ns 0 0 0 952 B
#7657 ExecuteAsync netcoreapp3.1 2.4μs 11.3ns 43.8ns 0 0 0 952 B
#7657 ExecuteAsync net472 2.59μs 1.12ns 4.17ns 0.142 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 6.81μs 9.91ns 37.1ns 0 0 0 2.36 KB
master SendAsync netcoreapp3.1 8.57μs 19.8ns 76.8ns 0 0 0 2.9 KB
master SendAsync net472 12.7μs 8.47ns 30.5ns 0.444 0 0 3.18 KB
#7657 SendAsync net6.0 6.78μs 2.29ns 8.26ns 0 0 0 2.36 KB
#7657 SendAsync netcoreapp3.1 8.5μs 18.8ns 72.7ns 0 0 0 2.9 KB
#7657 SendAsync net472 12.4μs 8.5ns 31.8ns 0.495 0 0 3.18 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7657

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 258.82 KB 277.65 KB 18.83 KB 7.28%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1 43.62 KB 44.04 KB 424 B 0.97%

Fewer allocations 🎉 in #7657

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472 286.72 KB 278.53 KB -8.19 KB -2.86%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 65.54 KB 57.34 KB -8.19 KB -12.50%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 45.5μs 244ns 2.06μs 0 0 0 43.82 KB
master StringConcatBenchmark netcoreapp3.1 48.8μs 241ns 1.28μs 0 0 0 43.62 KB
master StringConcatBenchmark net472 56.6μs 151ns 544ns 0 0 0 65.54 KB
master StringConcatAspectBenchmark net6.0 466μs 2.04μs 7.06μs 0 0 0 277.52 KB
master StringConcatAspectBenchmark netcoreapp3.1 514μs 1.57μs 5.66μs 0 0 0 258.82 KB
master StringConcatAspectBenchmark net472 400μs 1.7μs 6.38μs 0 0 0 286.72 KB
#7657 StringConcatBenchmark net6.0 45.6μs 256ns 2.1μs 0 0 0 43.8 KB
#7657 StringConcatBenchmark netcoreapp3.1 47.1μs 211ns 733ns 0 0 0 44.04 KB
#7657 StringConcatBenchmark net472 58.1μs 141ns 527ns 0 0 0 57.34 KB
#7657 StringConcatAspectBenchmark net6.0 509μs 1.12μs 5.24μs 0 0 0 277.1 KB
#7657 StringConcatAspectBenchmark netcoreapp3.1 511μs 6.06μs 59.7μs 0 0 0 277.65 KB
#7657 StringConcatAspectBenchmark net472 411μs 2.28μs 15.6μs 0 0 0 278.53 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.66μs 1.45ns 5.21ns 0 0 0 1.7 KB
master EnrichedLog netcoreapp3.1 3.63μs 16.3ns 63.3ns 0 0 0 1.7 KB
master EnrichedLog net472 3.72μs 4.81ns 18.6ns 0.259 0 0 1.64 KB
#7657 EnrichedLog net6.0 2.53μs 11.6ns 47.6ns 0 0 0 1.7 KB
#7657 EnrichedLog netcoreapp3.1 3.52μs 17.2ns 71ns 0 0 0 1.7 KB
#7657 EnrichedLog net472 3.81μs 4.19ns 15.7ns 0.244 0 0 1.64 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 123μs 91.6ns 343ns 0 0 0 4.31 KB
master EnrichedLog netcoreapp3.1 129μs 203ns 759ns 0 0 0 4.31 KB
master EnrichedLog net472 167μs 186ns 719ns 0 0 0 4.52 KB
#7657 EnrichedLog net6.0 122μs 47.1ns 176ns 0 0 0 4.31 KB
#7657 EnrichedLog netcoreapp3.1 129μs 130ns 468ns 0 0 0 4.31 KB
#7657 EnrichedLog net472 168μs 475ns 1.84μs 0 0 0 4.51 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.93μs 5.08ns 18.3ns 0 0 0 2.26 KB
master EnrichedLog netcoreapp3.1 6.89μs 12.1ns 46.8ns 0 0 0 2.26 KB
master EnrichedLog net472 7.62μs 10.8ns 41.8ns 0.306 0 0 2.08 KB
#7657 EnrichedLog net6.0 4.88μs 11.2ns 43.3ns 0 0 0 2.26 KB
#7657 EnrichedLog netcoreapp3.1 6.66μs 14.2ns 55ns 0 0 0 2.26 KB
#7657 EnrichedLog net472 7.51μs 5.45ns 20.4ns 0.299 0 0 2.08 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.96μs 1.96ns 7.61ns 0 0 0 1.2 KB
master SendReceive netcoreapp3.1 2.59μs 13.4ns 63.1ns 0 0 0 1.2 KB
master SendReceive net472 3.23μs 7.27ns 28.1ns 0.19 0 0 1.2 KB
#7657 SendReceive net6.0 1.97μs 10.3ns 49.2ns 0 0 0 1.2 KB
#7657 SendReceive netcoreapp3.1 2.65μs 10.9ns 42.1ns 0 0 0 1.2 KB
#7657 SendReceive net472 3.07μs 1.26ns 4.9ns 0.182 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 4.2μs 8.82ns 33ns 0 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 5.84μs 21ns 81.2ns 0 0 0 1.63 KB
master EnrichedLog net472 6.71μs 5.72ns 22.2ns 0.301 0 0 2.03 KB
#7657 EnrichedLog net6.0 4.27μs 5.2ns 19.4ns 0 0 0 1.58 KB
#7657 EnrichedLog netcoreapp3.1 5.33μs 17ns 65.7ns 0 0 0 1.63 KB
#7657 EnrichedLog net472 6.34μs 7.97ns 30.9ns 0.319 0 0 2.03 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 798ns 0.151ns 0.523ns 0 0 0 576 B
master StartFinishSpan netcoreapp3.1 946ns 4.52ns 19.2ns 0 0 0 576 B
master StartFinishSpan net472 898ns 0.629ns 2.44ns 0.0916 0 0 578 B
master StartFinishScope net6.0 922ns 3.14ns 11.7ns 0 0 0 696 B
master StartFinishScope netcoreapp3.1 1.17μs 5.99ns 26.8ns 0 0 0 696 B
master StartFinishScope net472 1.1μs 0.0874ns 0.339ns 0.0995 0 0 658 B
#7657 StartFinishSpan net6.0 737ns 0.292ns 1.05ns 0 0 0 576 B
#7657 StartFinishSpan netcoreapp3.1 912ns 4.38ns 17ns 0 0 0 576 B
#7657 StartFinishSpan net472 869ns 0.158ns 0.59ns 0.0916 0 0 578 B
#7657 StartFinishScope net6.0 896ns 4.79ns 26.2ns 0 0 0 696 B
#7657 StartFinishScope netcoreapp3.1 1.15μs 5.88ns 28.2ns 0 0 0 696 B
#7657 StartFinishScope net472 1.05μs 0.833ns 3.22ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 1.07μs 4.47ns 17.3ns 0 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 1.4μs 6.41ns 25.6ns 0 0 0 696 B
master RunOnMethodBegin net472 1.43μs 0.561ns 2.02ns 0.101 0 0 658 B
#7657 RunOnMethodBegin net6.0 1.05μs 5.09ns 19.7ns 0 0 0 696 B
#7657 RunOnMethodBegin netcoreapp3.1 1.41μs 2.13ns 8.26ns 0 0 0 696 B
#7657 RunOnMethodBegin net472 1.42μs 0.619ns 2.14ns 0.1 0 0 658 B

chrisnas
chrisnas approved these changes Oct 17, 2025
View reviewed changes
Copy link
Contributor

@chrisnas chrisnas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with maybe a renaming

profiler/src/ProfilerEngine/Datadog.Linux.ApiWrapper/functions_to_wrap.c
// This variable will be a pointer to that shared memory region.
__attribute__((visibility("hidden")))
atomic_int is_app_crashing = 0;
int* is_app_crashing = NULL;
Copy link
Contributor

@chrisnas chrisnas Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rename the variable to p_is_app_crashing to make it more explicit that it is a pointer

profiler/src/ProfilerEngine/Datadog.Linux.ApiWrapper/functions_to_wrap.c
// the parent process won't be notified that the app is crashing.
is_app_crashing = mmap(NULL, sizeof(int), PROT_READ | PROT_WRITE,
MAP_SHARED | MAP_ANONYMOUS, -1, 0);
if (is_app_crashing != MAP_FAILED) {
Copy link
Contributor

@chrisnas chrisnas Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we at least log a message because if it fails (even if it should not happen), we're doomed?

Copy link
Collaborator Author

@gleocadie gleocadie Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nop, at this point nothing is ready. That said, there is no logging from this library.

profiler/src/ProfilerEngine/Datadog.Linux.ApiWrapper/functions_to_wrap.c
#endif
// if we failed at allocating memory for the shared variable
// the parent process won't be notified that the app is crashing.
is_app_crashing = mmap(NULL, sizeof(int), PROT_READ | PROT_WRITE,
Copy link
Contributor

@chrisnas chrisnas Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know for linux but on Windows, objects need to be named in order to be shared between processes.
Just to be sure that I understand how this is supposed to work and why it does not matter for Linux:

  1. the parent code calls this init() function
  2. the forked child does not but inherits the value of this variable from its parent
  3. the forked child changes the value pointed to so that the parent could also see that change when calling dd_inside_wrapped_functions()

Copy link
Collaborator Author

@gleocadie gleocadie Oct 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as defined in the call to mmap, it's an anonymous area.

Just to be sure that I understand how this is supposed to work and why it does not matter for Linux:

I do not understand what you mean by it does not matter for Linux. What does not matter for Linux ?

@gleocadie gleocadie merged commit abe7838 into master Oct 17, 2025
160 checks passed
@gleocadie gleocadie deleted the gleocadie/fix-profiler-crashing-an-already-crashing-app branch October 17, 2025 19:52
@github-actions github-actions bot added this to the vNext-v3 milestone Oct 17, 2025
@kevingosse
Copy link
Contributor

kevingosse commented Oct 17, 2025

Good job but also wtf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chrisnas chrisnas chrisnas approved these changes

@andrewlock andrewlock andrewlock approved these changes

Assignees

No one assigned

Labels

area:crashtracking area:profiler Issues related to the continous-profiler identified-by:crashtracking

Projects

None yet

Milestone

3.29.0

Development

Successfully merging this pull request may close these issues.

4 participants

@gleocadie @kevingosse @chrisnas @andrewlock