Skip to content

http2: release queued data memory on stream cancellation - #67113

Merged
nodejs-github-bot merged 1 commit into
nodejs:mainfrom
inoway46:fix-http2-cancel-memory-accounting
Oct 31, 2026
Merged

nodejs-github-bot merged 1 commit into
nodejs:mainfrom
inoway46:fix-http2-cancel-memory-accounting

Conversation

@inoway46

Copy link
Copy Markdown
Contributor

Canceling a stream with queued outbound data leaves that data charged to maxSessionMemory. Subsequent requests on the same session can then fail even though the queued data has been discarded.

Release the remaining outbound memory accounting when cleaning up the stream's write queue, and add a regression test for session reuse after cancellation.

Found while investigating #66739. This does not change response-header admission or resolve the early-response issue itself.

Refs: #66739

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/http2
  • @nodejs/net

@nodejs-github-bot nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. http2 Issues and PRs related to the http2 subsystem. needs-ci PRs that need a full CI run. labels Oct 29, 2026
Release the memory charged for queued outbound data when a stream is
canceled, so subsequent requests can reuse the session's memory budget.

Refs: nodejs#66739
Assisted-by: Codex
Signed-off-by: inoway46 <inoueyuya416@gmail.com>
@inoway46
inoway46 force-pushed the fix-http2-cancel-memory-accounting branch from f8ee532 to 15bcc85 Compare October 29, 2026 16:27
@inoway46
inoway46 marked this pull request as ready for review October 29, 2026 16:29
@inoway46
inoway46 requested a review from pimterry October 29, 2026 16:30

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 29, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. Only starts once the PR has an approving review. label Oct 29, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 90.66%. Comparing base (98dbf60) to head (15bcc85).
⚠️ Report is 51 commits behind head on main.

Files with missing lines Patch % Lines
src/node_http2.cc 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #67113    +/-   ##
========================================
  Coverage   90.65%   90.66%            
========================================
  Files         797      797            
  Lines      278268   278793   +525     
  Branches    53517    53617   +100     
========================================
+ Hits       252261   252763   +502     
+ Misses      16386    16381     -5     
- Partials     9621     9649    +28     
Files with missing lines Coverage Δ
src/node_http2.cc 82.05% <50.00%> (-0.08%) ⬇️

... and 42 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@inoway46 inoway46 added the resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. label Oct 30, 2026
@github-actions github-actions Bot removed the resume-ci Add this label to resume the latest eligible Jenkins CI run on a PR with an approving review. label Oct 30, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@inoway46 inoway46 added author ready PRs with CI started, the required approvals, and no outstanding review comments. and removed needs-ci PRs that need a full CI run. labels Oct 30, 2026
@pimterry pimterry added the commit-queue PRs queued for automated landing through the Commit Queue. label Oct 30, 2026
@nodejs-github-bot
nodejs-github-bot merged commit 80a74e1 into nodejs:main Oct 31, 2026
94 checks passed
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Landed in 80a74e1

@nodejs-github-bot nodejs-github-bot removed the commit-queue PRs queued for automated landing through the Commit Queue. label Oct 31, 2026
inoway46 added a commit to inoway46/node that referenced this pull request Nov 1, 2026
Allow clients to receive responses when queued request data exceeds
maxSessionMemory, while preserving limits on received headers.

Fixes: nodejs#66739
Refs: nodejs#67113
Assisted-by: Codex
Signed-off-by: inoway46 <inoueyuya416@gmail.com>
fatihguzeldev added a commit to fatihguzeldev/node that referenced this pull request Nov 1, 2026
Signed-off-by: Fatih Güzel <fatihguzeldev@gmail.com>
@mishushakov

Copy link
Copy Markdown

Could this get backport-requested-v24.x / backport-requested-v22.x (and land in v25.x)? We are hitting the leak this fixes in production through undici's HTTP/2 client.

Impact. undici ≥ 8 multiplexes up to 100 streams on one long-lived Http2Session. Every stream that is destroyed before its queued body is flushed (request timeout, AbortSignal, peer RST_STREAM, GOAWAY mid-upload) permanently loses its queued bytes from the session's maxSessionMemory budget. Once the budget is gone, Http2Stream::AddHeader fails for every later response on that session and Node resets each of them locally with NGHTTP2_ENHANCE_YOUR_CALM (surfaced by fetch as TypeError: fetch failed, cause ERR_HTTP2_STREAM_ERROR). A busy process never lets the session go idle, so it never recovers.

Measured on v24.22.2 (undici 8.12.3, maxSessionMemory: 1 to shorten the repro):

before abort : 50/50 small POSTs ok
one 16 MiB POST aborted after 300 ms
after abort  : 0/50 ok — every request RST(ENHANCE_YOUR_CALM), still 0/50 two seconds later

With the default 10 MB budget a single aborted 16 MiB upload loses 2–5 MB of it for the session's lifetime (checked by bisecting maxSessionMemory: 2 → poisoned, 5 → not).

Minimal repro (no undici, plain http2):

const http2 = require('node:http2')
const session = http2.connect('https://<any-h2-origin-accepting-uploads>', { maxSessionMemory: 1 })

function post (body, ms) {
  return new Promise((resolve) => {
    const req = session.request({ ':method': 'POST', ':path': '/upload' })
    req.on('response', (h) => resolve({ status: h[':status'] }))
    req.on('close', () => resolve({ rstCode: req.rstCode }))
    req.end(body)
    if (ms) setTimeout(() => req.destroy(), ms) // abort while body is still queued
  })
}

;(async () => {
  console.log(await post(Buffer.alloc(16 << 20), 300))      // { rstCode: 8 } CANCEL, expected
  console.log(await post(Buffer.alloc(4096)))                // { rstCode: 11 } ENHANCE_YOUR_CALM — leaked budget
  console.log(await post(Buffer.alloc(4096)))                // { rstCode: 11 } forever
})()

Context: e2b-dev/e2b SDK-425 — concurrent files.write() failing with ENHANCE_YOUR_CALM after moving from undici 7 (one stream per session) to undici 8 (100 streams per session).

@inoway46

inoway46 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed report! Given the production impact, I agree this is worth backporting to the supported LTS lines.

@inoway46 inoway46 added lts-watch-v22.x PRs that may need to be released in v22.x lts-watch-v24.x PRs that may need to be released in v24.x lts-watch-v26.x PRs that may need to be released in v26.x. labels Nov 3, 2026
mishushakov added a commit to e2b-dev/E2B that referenced this pull request Nov 3, 2026
…TP/2 dispatchers (#2086)

## Summary

Fixes [SDK-425](https://linear.app/e2b/issue/SDK-425): concurrent
`sandbox.files.write()` calls fail with `TypeError: fetch failed` →
`NGHTTP2_ENHANCE_YOUR_CALM` since `e2b` ≥ 2.35 (undici 8). Simpler
alternative to #2083 (byte budget + one-off sessions), same
measurements.

**Cause.** The RST is sent by Node's own http2 client, not by E2B infra.
Node queues each buffered request body in full against the session's 10
MB `maxSessionMemory`; if a response HEADERS frame arrives on *any*
stream while that budget is exhausted, Node resets that stream with
`ENHANCE_YOUR_CALM`. undici 7 kept one request per session (so our
10-connection pool = 10 budgets); undici 8 multiplexes, so every envd
REST request in the process shares one session and one budget — parallel
or large uploads reset each other and the reads next to them. Streams
destroyed before their body is flushed (timeouts, aborts, killed
sandboxes) additionally leak their queued bytes on Node ≤ 24 (fixed on
`main` by nodejs/node#67113), so a session degrades until every small
write on it fails — the customer's "climbing since Monday".

**Fix (JS SDK only — Python uses pyqwest/hyper, not Node http2):**

1. `Filesystem` POSTs `/files` (octet-stream, streamed, gzip and
multipart uploads) through `createEnvdUploadFetch` — an HTTP/1.1
dispatcher pool of its own (`E2B_ENVD_UPLOAD_CONNECTIONS`, default 50,
like `E2B_ENVD_RPC_CONNECTIONS`); everything else (`read`, `makeDir`,
RPC, health…) stays on HTTP/2, and `httpVersion` / `E2B_HTTP_VERSION`
keep governing those. The `ConnectionOpts.httpVersion` JSDoc documents
the exception.
   ```ts
   // sandbox/filesystem/index.ts
const uploadFetch = createEnvdUploadFetch({ proxy:
connectionConfig.proxy })
   …
   this.envdApi.api.POST('/files', { …, fetch: this.uploadFetch })
   ```
One request per connection, no shared memory budget — what undici 7
effectively gave 2.27.1 (which capped uploads at 10 per process; this
pool allows 50, beyond that uploads queue in undici with their request
timeout running).
2. `buildDispatchedFetch` (`src/undici.ts`): a request that fails with
`ERR_HTTP2_STREAM_ERROR` +
`NGHTTP2_ENHANCE_YOUR_CALM`/`NGHTTP2_INTERNAL_ERROR`
(`isPoisonedHTTP2SessionError`) retires that dispatcher and builds a
fresh one for subsequent requests. The failed request is **not**
replayed (envd may already have applied the write). Only the dispatcher
that served the failing request can replace the current one. Safety net
for whatever else might fill a session.

No API change.

**Measured** (built package, real sandboxes, Node 24.22; ALPN confirmed:
writes `http/1.1`, reads `h2`):

| scenario | 2.62.1 | this PR |
|---|---|---|
| 10 × 2 MB parallel writes, 5 rounds | 16/50 failed | 0/50 |
| 30 × 1 MB, 20 × 1 MB, 4 × 16 MB, customer pattern (makeDir + 10
parallel small writes ×200) | — | 0 failed |
| 16 MB write + 10 concurrent `files.read()` | 30/30 reads failed | 0/33
|
| poisoned session (`maxSessionMemory=1`, aborted big writes) | 50/50
fail, forever | 0 failed |
| streamed (async iterable), gzip, multipart, string, aborted upload | —
| all OK |

**Upstream:** backport request on nodejs/node#67113
(nodejs/node#67113 (comment)) and
undici issue nodejs/undici#6149.

Link to Devin session:
https://app.devin.ai/sessions/98f09d080013403b954d3f7b07c51b3b
Open in Devin Desktop:
https://app.devin.ai/desktop/session/98f09d080013403b954d3f7b07c51b3b?variant=devin
Requested by: @mishushakov

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mish@e2b.dev <mish@e2b.dev>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author ready PRs with CI started, the required approvals, and no outstanding review comments. c++ Issues and PRs that require attention from people who are familiar with C++. http2 Issues and PRs related to the http2 subsystem. lts-watch-v22.x PRs that may need to be released in v22.x lts-watch-v24.x PRs that may need to be released in v24.x lts-watch-v26.x PRs that may need to be released in v26.x.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants