module: rework of memory management in vm APIs with the importModuleDynamically option - #48510
module: rework of memory management in vm APIs with the importModuleDynamically option#48510joyeecheung wants to merge 3 commits into
vm APIs with the importModuleDynamically option#48510Conversation
|
Review requested:
|
nodejs-github-bot
added
lib / src
joyeecheung
changed the title
|
Still need to do a bit more investigation to figure out it's really safe this time around, so marking it as WIP (I've done some walkthrough of the |
joyeecheung
changed the title
|
amazing! |
|
It looks to me like symbols-as-weakmap-keys is now Stage 4 as of 2023-01-30, but not all of the documents have been updated. I would guess back-porting this fix to Node 16 or 18 is going to be a challenge though, right? |
1751348 to
76f5ce9
Compare
Nice. Then I think we don't have to flag this for v20. But yeah v18 or v16 didn't implement that proposal (and we probably shouldn't backport it / cannot do a full upgrade because LTS) so it cannot be backported. I've tweaked the registry object a bit, added tests, and some analysis to the commit messages about why it fixes the leaks/segfaults. Although I think for the |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
Notable changes: deps: * add v8::Object::SetInternalFieldForNodeCore() (Joyee Cheung) #49874 doc: * deprecate `fs.F_OK`, `fs.R_OK`, `fs.W_OK`, `fs.X_OK` (Livia Medeiros) #49683 * deprecate `util.toUSVString` (Yagiz Nizipli) #49725 * deprecate calling `promisify` on a function that returns a promise (Antoine du Hamel) #49647 esm: * set all hooks as release candidate (Geoffrey Booth) #49597 module: * fix the leak in SourceTextModule and ContextifySript (Joyee Cheung) #48510 * fix leak of vm.SyntheticModule (Joyee Cheung) #48510 * use symbol in WeakMap to manage host defined options (Joyee Cheung) #48510 src: * (SEMVER-MINOR) allow embedders to override NODE_MODULE_VERSION (Cheng Zhao) #49279 stream: * use bitmap in writable state (Raz Luvaton) #49834 * use bitmap in readable state (Benjamin Gruenbaum) #49745 * improve webstream readable async iterator performance (Raz Luvaton) #49662 test_runner: * (SEMVER-MINOR) accept `testOnly` in `run` (Moshe Atlow) #49753 * (SEMVER-MINOR) add junit reporter (Moshe Atlow) #49614 PR-URL: #49932
Notable changes: deps: * add v8::Object::SetInternalFieldForNodeCore() (Joyee Cheung) #49874 doc: * deprecate `fs.F_OK`, `fs.R_OK`, `fs.W_OK`, `fs.X_OK` (Livia Medeiros) #49683 * deprecate `util.toUSVString` (Yagiz Nizipli) #49725 * deprecate calling `promisify` on a function that returns a promise (Antoine du Hamel) #49647 esm: * set all hooks as release candidate (Geoffrey Booth) #49597 module: * fix the leak in SourceTextModule and ContextifySript (Joyee Cheung) #48510 * fix leak of vm.SyntheticModule (Joyee Cheung) #48510 * use symbol in WeakMap to manage host defined options (Joyee Cheung) #48510 src: * (SEMVER-MINOR) allow embedders to override NODE_MODULE_VERSION (Cheng Zhao) #49279 stream: * use bitmap in writable state (Raz Luvaton) #49834 * use bitmap in readable state (Benjamin Gruenbaum) #49745 * improve webstream readable async iterator performance (Raz Luvaton) #49662 test_runner: * (SEMVER-MINOR) accept `testOnly` in `run` (Moshe Atlow) #49753 * (SEMVER-MINOR) add junit reporter (Moshe Atlow) #49614 PR-URL: #49932
|
@joyeecheung Is there a way to detect this fix from unprivileged code that is better than checking a bunch of version numbers? |
13 participants
This PR (hopefully) fixes a bunch of memory leaks & use-after-free surrounding
vm.Script,vm.compileFunction,vm.SyntheticModuleandvm.SourceTextModulewhen dynamic import is used (see the issues referenced below) and makes it possible for people to upgrade from older versions of Node.js.This cannot land as-is on v20.x as it depends on #49491 and #49419 which break the ABI on v20x. We need a special Node.js v20-only extension to the V8 API for this to work without breaking the ABI.
module: use symbol in WeakMap to manage host defined options
Previously when managing the importModuleDynamically callback of
vm.compileFunction(), we use an ID number as the host defined option
and maintain a per-Environment ID -> CompiledFnEntry map to retain
the top-level referrer function returned by vm.compileFunction() in
order to pass it back to the callback, but it would leak because with
how we used v8::Persistent to maintain this reference, V8 would not
be able to understand the cycle and would just think that the
CompiledFnEntry was supposed to live forever. We made an attempt
to make that reference known to V8 by making the CompiledFnEntry weak
and using a private symbol to make CompiledFnEntry strongly
references the top-level referrer function in
#46785, but that turned out to be
unsound, because the there's no guarantee that the top-level function
must be alive while import() can still be initiated from that
function, since V8 could discard the top-level function and only keep
inner functions alive, so relying on the top-level function to keep
the CompiledFnEntry alive could result in use-after-free which caused
a revert of that fix.
With this patch we use a symbol in the host defined options instead of
a number, because with the stage-3 symbol-as-weakmap-keys proposal
we could directly use that symbol to keep the referrer alive using a
WeakMap. As a bonus this also keeps the other kinds of referrers
alive as long as import() can still be initiated from that
Script/Module, so this also fixes the long-standing crash caused by
vm.Script being GC'ed too early when its importModuleDynamically
callback still needs it.
module: fix leak of vm.SyntheticModule
Previously we maintain a strong persistent reference to the
ModuleWrap to retrieve the ID-to-ModuleWrap mapping from
the HostImportModuleDynamicallyCallback using the number ID
stored in the host-defined options. As a result the ModuleWrap
would be kept alive until the Environment is shut down, which
would be a leak for user code. With the new symbol-based
host-defined option we can just get the ModuleWrap from the
JS-land WeakMap so there's now no need to maintain this
strong reference. This would at least fix the leak for
vm.SyntheticModule. vm.SourceTextModule is still leaking
due to the strong persistent reference to the v8::Module.
module: fix the leak in SourceTextModule and ContextifySript
Replace the persistent handles to v8::Module and
v8::UnboundScript with an internal reference that V8's GC is
aware of to fix the leaks.
Refs: #44211
Refs: #42080
Refs: #47096
Refs: #43205
Refs: #38695