fix(v8-bridge): yield a macrotask between bridge socket reads (undici keep-alive leak)#122
Merged
Merged
Conversation
|
🚅 Deployed to the secure-exec-pr-122 environment in rivet-frontend
🚅 Deployed to the secure-exec-pr-122 environment in secure-exec
|
_pumpBridgeReads drained the synchronous _netSocketReadRaw in a tight loop and emitted readable/data for an entire HTTP response in one synchronous burst. That collapses the event-loop turn boundaries undici's keep-alive socket recycling depends on: onMessageComplete resolves the caller's fetch synchronously and only defers reuse via setImmediate(client[kResume]), so the caller's microtask dispatches the next request before kResume runs. Every Client stays kNeedDrain, the pool allocates a fresh Client+socket per request, and each registers EventEmitter listeners until the in-VM Node process dies (MaxListenersExceededWarning + unbounded memory / OOM). Await a setImmediate-backed macrotask yield before delivering each payload so socket bytes surface across distinct event-loop turns, exactly as they do on real Node where each readable arrives in its own I/O callback. This lets setImmediate(kResume) clear kNeedDrain before the next dispatch, so the pool reuses one keep-alive socket. Empirically: 40 sequential requests went from 40 sockets / 41 Clients to 1 socket / 2 Clients. Adds a deterministic regression test (javascript_v8 suite) that scripts two socket chunks over the mocked net bridge and asserts a setImmediate scheduled when the first chunk lands interleaves before the second chunk. It fails on the synchronous-burst trace (data:chunk-1,data:chunk-2,immediate) and passes on the fixed trace (data:chunk-1,immediate,data:chunk-2). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
a27574f to
4f6f4d0
Compare
NathanFlurry
added a commit
that referenced
this pull request
Jun 25, 2026
…idge leak Three native-sidecar fixes behind long-lived VM latency/stall. 1) Read-side shadow-tree re-walk Read-side guest fs ops (Exists/Stat/Lstat/ReadFile/Pread) reconcile the host shadow tree into the kernel VFS, re-reading+re-writing EVERY file on EVERY op (O(whole tree), super-linear). Add an rsync-style (size,mode,mtime) lstat skip for unchanged files. Test read_side_ops_skip_unchanged_shadow_files: warm read over an unchanged 800-file tree >4x cheaper than cold (cold=22.7s, warm=28ms debug). 2) EventEmitter shim: spurious MaxListenersExceededWarning ensureEventEmitterInitialized() defaulted _maxListenersWarned but not _maxListeners, so emitters that acquire _events outside our ctor (undici's Client/Pool/Agent) had _maxListeners=undefined and 'total <= undefined' warned on the FIRST listener. Default _maxListeners so the threshold is meaningful. 3) undici client-per-request leak (the real net-bridge leak) The bridge's UndiciAgent was created with an UNBOUNDED per-origin pool. Requests that overlap while clients are still connecting each find every client kNeedDrain and spawn a fresh Client+socket -- and for HTTPS the LLM path is HTTP/2 (ALPN), so each spawn is a whole new h2 session. The synchronous bridge reads widen that connect window (the #122 per-payload macrotask yield only helps h1 same-socket reuse, nothing for the h2 connect-window herd). Over a long many-call turn the abandoned clients accumulate connect/close/drain/error/finish/readable/end/ terminated listeners without bound -> http2 degradation -> 'Request was aborted.' Bound connections (6, browser-like; h2 multiplexes within each) so excess requests queue on existing clients instead of spawning new ones. Test keepalive_no_listener_leak now drives CONCURRENT requests (the trigger) and asserts the host-side connection count stays bounded: 160 requests over 6 connections with the cap vs 16 (2x concurrency) without it. A process.on('warning') handler surfaces any MaxListenersExceededWarning to stderr (the warning alone is insufficient -- leaked listeners spread across per-client emitters). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NathanFlurry
added a commit
that referenced
this pull request
Jun 25, 2026
…idge leak Three native-sidecar fixes behind long-lived VM latency/stall. 1) Read-side shadow-tree re-walk Read-side guest fs ops (Exists/Stat/Lstat/ReadFile/Pread) reconcile the host shadow tree into the kernel VFS, re-reading+re-writing EVERY file on EVERY op (O(whole tree), super-linear). Add an rsync-style (size,mode,mtime) lstat skip for unchanged files. Test read_side_ops_skip_unchanged_shadow_files: warm read over an unchanged 800-file tree >4x cheaper than cold (cold=22.7s, warm=28ms debug). 2) EventEmitter shim: spurious MaxListenersExceededWarning ensureEventEmitterInitialized() defaulted _maxListenersWarned but not _maxListeners, so emitters that acquire _events outside our ctor (undici's Client/Pool/Agent) had _maxListeners=undefined and 'total <= undefined' warned on the FIRST listener. Default _maxListeners so the threshold is meaningful. 3) undici client-per-request leak (the real net-bridge leak) The bridge's UndiciAgent was created with an UNBOUNDED per-origin pool. Requests that overlap while clients are still connecting each find every client kNeedDrain and spawn a fresh Client+socket -- and for HTTPS the LLM path is HTTP/2 (ALPN), so each spawn is a whole new h2 session. The synchronous bridge reads widen that connect window (the #122 per-payload macrotask yield only helps h1 same-socket reuse, nothing for the h2 connect-window herd). Over a long many-call turn the abandoned clients accumulate connect/close/drain/error/finish/readable/end/ terminated listeners without bound -> http2 degradation -> 'Request was aborted.' Bound connections (6, browser-like; h2 multiplexes within each) so excess requests queue on existing clients instead of spawning new ones. Test keepalive_no_listener_leak now drives CONCURRENT requests (the trigger) and asserts the host-side connection count stays bounded: 160 requests over 6 connections with the cap vs 16 (2x concurrency) without it. A process.on('warning') handler surfaces any MaxListenersExceededWarning to stderr (the warning alone is insufficient -- leaked listeners spread across per-client emitters). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
NathanFlurry
added a commit
that referenced
this pull request
Jun 25, 2026
…idge leak (#128) Three native-sidecar fixes behind long-lived VM latency/stall. 1) Read-side shadow-tree re-walk Read-side guest fs ops (Exists/Stat/Lstat/ReadFile/Pread) reconcile the host shadow tree into the kernel VFS, re-reading+re-writing EVERY file on EVERY op (O(whole tree), super-linear). Add an rsync-style (size,mode,mtime) lstat skip for unchanged files. Test read_side_ops_skip_unchanged_shadow_files: warm read over an unchanged 800-file tree >4x cheaper than cold (cold=22.7s, warm=28ms debug). 2) EventEmitter shim: spurious MaxListenersExceededWarning ensureEventEmitterInitialized() defaulted _maxListenersWarned but not _maxListeners, so emitters that acquire _events outside our ctor (undici's Client/Pool/Agent) had _maxListeners=undefined and 'total <= undefined' warned on the FIRST listener. Default _maxListeners so the threshold is meaningful. 3) undici client-per-request leak (the real net-bridge leak) The bridge's UndiciAgent was created with an UNBOUNDED per-origin pool. Requests that overlap while clients are still connecting each find every client kNeedDrain and spawn a fresh Client+socket -- and for HTTPS the LLM path is HTTP/2 (ALPN), so each spawn is a whole new h2 session. The synchronous bridge reads widen that connect window (the #122 per-payload macrotask yield only helps h1 same-socket reuse, nothing for the h2 connect-window herd). Over a long many-call turn the abandoned clients accumulate connect/close/drain/error/finish/readable/end/ terminated listeners without bound -> http2 degradation -> 'Request was aborted.' Bound connections (6, browser-like; h2 multiplexes within each) so excess requests queue on existing clients instead of spawning new ones. Test keepalive_no_listener_leak now drives CONCURRENT requests (the trigger) and asserts the host-side connection count stays bounded: 160 requests over 6 connections with the cap vs 16 (2x concurrency) without it. A process.on('warning') handler surfaces any MaxListenersExceededWarning to stderr (the warning alone is insufficient -- leaked listeners spread across per-client emitters). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The in-VM bridge socket read loop (
_pumpBridgeReadsincrates/execution/assets/v8-bridge.source.js) isasyncbut contained noawait. Because_netSocketReadRawis synchronous, the loop drained every available byte and emittedreadable/datafor an entire HTTP response in one synchronous burst.That collapses the event-loop turn boundaries undici's keep-alive socket recycling depends on:
onMessageCompletein the same synchronous stack.fetchsynchronously, and only defers socket reuse withsetImmediate(client[kResume]).await fetch()continuation (a microtask) dispatches the next request before thatsetImmediatemacrotask runs.Clientis stillkNeedDrain, soPool.kGetDispatcherconstructs a new Client + socket per request.Each new Client registers ~4 EventEmitter listeners, so the count climbs until it trips
MaxListenersExceededWarningand the in-VM Node process exhausts memory and is OOM-killed (the V8 isolate's 128 MiB heap cap firesnear_heap_limit_callback→terminate_execution). It only reproduces in the VM because real Node delivers response bytes across multiple I/O turns, sosetImmediate(kResume)reliably fires before the next request.Fix
awaitasetImmediate-backed macrotask yield before delivering each payload, so socket bytes surface across distinct event-loop turns — exactly as on real Node where eachreadablearrives in its own I/O callback. This letssetImmediate(kResume)clearkNeedDrainbefore the next dispatch, so the pool reuses one keep-alive socket.Empirically (from the original investigation): 40 sequential requests went from 40 sockets / 41 Clients to 1 socket / 2 Clients.
The single
v8-bridge.source.jsfeeds both theexecutionandv8-runtimecrates, so both pick up the fix from one source.Test
Adds a deterministic regression test to the
javascript_v8suite that scripts two socket chunks over the mocked net bridge and schedules asetImmediate(standing in forsetImmediate(kResume)) the instant the first chunk lands. It asserts the macrotask interleaves before the second chunk:data:chunk-1,data:chunk-2,immediate→ guest throws → test fails.data:chunk-1,immediate,data:chunk-2→ test passes.Verified both directions locally (reverting the source edit reproduces the failure).