[EventPipe] AV at EventPipeInternal_GetNextEvent #122630
Description
Description
Some users that have been leveraging an EventListener to listen to basic GC events have been experiencing crashes due to an AV at
runtime/src/coreclr/vm/eventpipeinternal.cpp
Line 251 in b2969ed
Reproduction Steps
Unclear how to repro. The users were using .NET 8 and seem to just be using a standard EventListener to listen to events. The users mentioned that this crash seems to only occur in machines that have had their OS patched with kernel upgrades.
Expected behavior
EventPipe doesn't crash the runtime and doesn't return an invalid EventPipeEventInstance.
Actual behavior
Crashes due to AV
Regression?
No response
Known Workarounds
Disable EventListeners, use alternative APIs to grab the data held by the events, like GCMemoryInfo or other GC APIs for GC events.
Configuration
.NET 8, Linux-x64, kernel: 5.15.0-140-fips, AzureLinux3.0
Other information
While investigating the crash dump, we've observed that there is an EventPipeBuffer with it's buffer region all 0'd out, even though the EventPipeBuffer still needed its first EventPipeEventInstance read from its buffer region. (below current_read_event is at buffer)
0:224> ?? ((EventPipeSession*)sessionID)->buffer_manager->current_buffer
EventPipeBuffer * 0x00007f5e`280bfae0
+0x000 creation_timestamp : 0n1876226656953887
+0x008 writer_thread : 0x00007f5e`28000b40 _EventPipeThread
+0x010 buffer : 0x00007fbe`fcd89000 -> 0 ''
+0x018 current : 0x00007fbe`fcd89f78 -> 0 ''
+0x020 limit : 0x00007fbe`fcda2000 -> ??
+0x028 current_read_event : 0x00007fbe`fcd89000 _EventPipeEventInstance
+0x030 prev_buffer : (null)
+0x038 next_buffer : 0x00007f5e`280fc890 _EventPipeBuffer
+0x040 state : 1
+0x044 event_sequence_number : 0x2edc13
This is unexpected, as buffer regions should only be cleared when
- Buffer allocation failed - This buffer shouldn't have been added to the buffer list
- Buffer manager failed to add the buffer to list - The buffer wouldn't have been added to the buffer list
- While advancing to a non-empty buffer, the current buffer's read event is NULL. From the dump, it's not NULL and is still the first EventPipeEventInstance.
Moreover, the thread that was logging to this buffer had two buffers in its list, the readable current_buffer above, and another writable buffer whose event sequence number was 0x2edc1c, indicating 9 events (0x2edc1c - 0x2edc13) being written in current_buffer above. The writable buffer also had valid non-zeroed EventPipeEventInstances in its buffer region.
With insufficient diagnostics to determine what exactly led to the EventPipeBuffer's buffer region being 0'd out, coupled with the dump suggesting a scenario that doesn't seem consistent with EventPipe logic, it's difficult to tell if this crash was really due to a race condition leading to use-after-free, or some external process corrupting memory.
