feat(benchmarks): full ecosystem command rows (spec 4.1)#219
Merged
NathanFlurry merged 8 commits intoJul 2, 2026
Merged
Conversation
…s via event The sidecar drains process output before emitting process_exited and the frame stream + event pump are FIFO, so once the exit event is observed no trailing output can follow it; the host-side quiet-turn drain (2 turns at 10ms) only remains for the snapshot-poll fallback exit path. Also stop awaiting signal-state refreshes on the exec critical path (start + first output) — kill paths await the in-flight refresh instead — and clean up the parked refresh in finishProcess so fallback exits release it too. Warm host-driven exec p50: node -e '' 30ms -> ~17ms, wasm true 52ms -> 39ms; whole wasm-command-floor lane down ~13ms/row; bench:gate green. New regression test: 10 sequential fast-exit 64KiB stdout captures.
Bake the constant wasm runner + wasi shim (~300KB of JS previously recompiled per exec) into the per-process V8 userland snapshot via guest_runtime.snapshot_userland_code, keyed and cached process-wide. Mode env AGENTOS_WASM_SNAPSHOT_RUNNER=auto|block|off: auto probes the snapshot cache without blocking (async warm kicked once per process) and falls back to the byte-identical inline runner until ready, so the cold path is unchanged. The sync-RPC glue is re-evaluated per exec (the snapshot-baked copy cannot bind session bridge fns). Module bytes are now base64-encoded once per module in a bounded, fingerprint-validated cache (64 entries, warn on evict, debug log reports per-entry and cumulative cache bytes) instead of fs::read+encode per exec. Adds AGENTOS_V8_SESSION_PHASES timers (snapshot_get / blob_clone / isolate_new / user_code_execute) and a measured NOTE against EagerCompile at snapshot build (moves cost to isolate deserialize). Warm wasm floor p50: true 39->33ms, pwd 48->43ms, ls-empty 81->75ms, date-version 127->115ms (52/62/96/140 before PR #212); cold first-exec improved; wasm suite green; bench:gate green. Remaining floor is isolate-per-exec + module decode — pooling tracked as Stage C.
The filtered install (--filter @secure-exec/benchmarks...) omitted packages/build-tools, so every bench-gate run died in the v8-bridge build script (missing Node dependencies) before gating anything — the PR gate has never actually run green on CI.
Pre-created V8 session workers (thread + snapshot-restored isolate, keyed by snapshot digest + heap limit) are claimed at session create via a warm hint, taking isolate_new (~4.2ms) and blob_clone off the per-exec critical path. Workers are never reused across guests (each exec still gets a virgin isolate); a wrong hint falls back to the existing lazy-create path. Capacity per key via AGENTOS_V8_WARM_ISOLATES (default 2, 0 disables); refill after claim; pool transitions logged; warm_worker_hit/miss counters in the session phases output. Seeding is deliberately conservative: auto mode stays snapshot-only and only AGENTOS_WASM_SNAPSHOT_RUNNER=block seeds workers (wasm-runner key + default node key). Reason: creating isolates on background threads while other isolates execute guest WebAssembly deterministically SIGSEGVs the pinned V8 130 (WasmCodePointerTable::AllocateUninitializedEntry inside Isolate::New, even under the isolate lifecycle lock — the race is against V8-internal wasm threads). Backtraces + notes in the rusty-v8-upgrade todo; enabling default seeding rides that upgrade. Warm floors (block, this host): node -e '' 17 -> 14.4ms p50, wasm true 33 -> ~30ms (warm_worker_hit 64/67, isolate_new down to 3 background calls); pwd 42.6 -> 37.5ms, printf-0b 44.7 -> 40.2ms, date 115 -> 111ms. wasm suite 8.9s (was 96s, pool+snapshot amortize debug isolates); v8-runtime tests and bench:gate green.
Module bytes now ride the in-process runtime protocol as Arc<Vec<u8>> from a raw-bytes cache (same bounded/fingerprinted semantics as the base64 cache it replaces) and are injected pre-exec as a __agentOSWasmModuleBytes Uint8Array; the runner prefers it over the AGENTOS_WASM_MODULE_BASE64 env fallback, so the per-exec base64 encode/decode (4.3ms @267KB, ~23ms @2.5MB) is gone in both snapshot and inline modes. The userland (wasm-runner) snapshot script is now compiled with EagerCompile: the fatter-blob isolate cost it causes is prepaid by parked warm workers, so the runner function no longer lazy-compiles inside user_code_execute. Warm floors (block, this host): wasm true 30 -> 19.7ms p50 (52ms at the start of the 3.2 campaign), echo hello 24.0ms, pwd 29.3ms, ls-empty ~39ms (was 96), date --version 30.4ms (was 140 — module-size tax eliminated), printf-64k 136ms (stdout streaming path, separate finding). Injection failure fails the exec loudly. wasm + v8-runtime suites green; bench:gate green (require_100_small flaked 4.3x once and passed 1.13x on rerun — bimodal, pre-existing, tracked).
WASM/shell commands write the kernel VFS, but a JS guest's mapped-shadow read arms returned the shadow's answer — including ENOENT — without materializing kernel state: readdir listed only shadow entries, exists and readlink probed only the shadow. Anything a wasm command created was invisible to node (fs.readdirSync ENOENT on a directory the client API could list). readdir now materializes the path from the kernel, merges kernel children into the shadow listing by name (shadow wins), and tolerates kernel ENOENT/ENOTDIR for shadow-only dirs; exists/readlink materialize first like fs.access always did. Mapped unlink/rmdir now mirror the removal into the kernel (best-effort, ENOENT tolerated) — the shadow cannot express deletions, so the new merge would otherwise resurrect a kernel-backed file in the very listing that follows the unlink. Five service regressions: wasm-mkdir visible to node readdir+stat, shell mkdir+redirect children merged, shadow/kernel union lists both writers' files exactly once, same-process shadow readdir, and unlink-no-resurrect (same process + next process). Discriminator repro (direct mkdir / shell mkdir / node mkdir / shell redirect) all visible; wasm suite green; focused readdir lane runs again (was ENOENT on its shell-made fixtures); bench:gate green (readdir_small +0.05ms from the kernel merge — 3.3 targets that row).
readdir (spec 3.3): the mapped-shadow arm now derives is_directory from the getdents d_type (DirEntry::file_type) instead of an anchored openat2 + statx per entry (symlinks keep the resolving fallback so symlink-to-dir still reports a directory), drops its redundant materialize-from-kernel side effect (the kernel union merge subsumes it for listing; opens still materialize), and the response crosses as one raw status-2 buffer ([kind][u32le len][name] per entry) decoded directly in the bridge instead of JSON->CBOR->V8 objects. fs/readdir_small guest 0.23 -> ~0.09ms p50 (4.5x node; Accept <=0.1ms met); readdir_big 0.57ms. promises.readdir stays on the CBOR path. streams (spec 3.4): new _fsWritevRaw / fs.writevSync raw multi-buffer RPC ([u32 count] + per-buffer [u32 len][bytes], sequential-write semantics), and WriteStream's end-flush sends its buffered chunks as ONE writev instead of an RPC per chunk. fs/stream_copy_big guest 7.6 -> 4.3ms, stream_copy_small 1.03 -> 0.63ms. The <=3ms Accept is NOT met — measured floor: the residual is sync-bridge byte throughput (~1.5ms per MiB per direction; a 1MiB copy is 2 large transfers), proven by an RPC-count probe (read-ahead batching to 1 RPC per direction made it SLOWER from extra copies and was reverted; the probe artifacts and the rejected round-2 read-ahead live in ~/progress/secure-exec/2026-07-02-*). Bridge byte-throughput is filed as the Phase 5 unblocker (same root cause as the stdout 1.3ms/KiB-chunk finding). Service regressions: raw readdir Dirent semantics (incl. symlink cases), writev stream-copy order, all five mapped-shadow tests; bench:gate green.
Every agent-facing command now has a verified host-vs-VM row: sh_pipeline, cd_tmp_pwd, echo_hello, cat_small/big, grep_small/big, sed_substitution, find_1000, tar_small/big, gzip_small/big, jq_extract joining ls_100 and git_init_commit — 16 rows, zero skips, every row asserts output correctness (match counts, extract-diff round trips, exact stdout, sha shape). Vendored wasm command binaries are validated at startup and missing ones fail loudly. Two-tier fixtures on the size-sensitive commands per the 1.5 convention; memory column included. First release-build numbers (warm vmCmd p50 vs host): echo 29ms/11x, cd 51ms/14x, cat 42ms/18x, sed 49ms/23x, jq 102ms/6.5x, git 1.17s/47x, gzip_big 629ms/54x, tar_big 5.1s/631x, grep_big 7.7s/961x, find_1000 5.3s/1470x — the Phase 5.2 worklist, now measured honestly.
This was referenced Jul 2, 2026
Member
Author
This was referenced Jul 2, 2026
NathanFlurry
changed the base branch from
stack/perf-fs-readdir-raw-fast-path-stream-writev-coalescing-js-lane-xlprnvwv
to
main
NathanFlurry
deleted the
stack/feat-benchmarks-full-ecosystem-command-rows-spec-4-1-soqurlqr
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every agent-facing command now has a verified host-vs-VM row: sh_pipeline,
cd_tmp_pwd, echo_hello, cat_small/big, grep_small/big, sed_substitution,
find_1000, tar_small/big, gzip_small/big, jq_extract joining ls_100 and
git_init_commit — 16 rows, zero skips, every row asserts output correctness
(match counts, extract-diff round trips, exact stdout, sha shape). Vendored
wasm command binaries are validated at startup and missing ones fail loudly.
Two-tier fixtures on the size-sensitive commands per the 1.5 convention;
memory column included. First release-build numbers (warm vmCmd p50 vs host):
echo 29ms/11x, cd 51ms/14x, cat 42ms/18x, sed 49ms/23x, jq 102ms/6.5x,
git 1.17s/47x, gzip_big 629ms/54x, tar_big 5.1s/631x, grep_big 7.7s/961x,
find_1000 5.3s/1470x — the Phase 5.2 worklist, now measured honestly.