OOM crash in scanJsonlFile / parseSessionFile: readViaStream loads entire file into memory despite using streams

[maucher]

Bug

Running codeburn on a large session corpus crashes with:

<--- Last few GCs --->

[91884:0x22ae7960]   151700 ms: Scavenge 4072.1 (4095.2) -> 4071.5 (4103.2) MB, 0.79 / 0.00 ms  (average mu = 0.751, current mu = 0.737) allocation failure;
[91884:0x22ae7960]   151762 ms: Mark-Compact (reduce) 4102.0 (4133.7) -> 4094.4 (4115.1) MB, 29.36 / 0.02 ms  (+ 6.6 ms in 1 steps since start of marking, biggest step 6.6 ms, walltime since start of marking 62 ms) (average mu = 0.657, current mu = 0.606)

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xb82c78 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0xeefa80 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0xeefd67 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x1101905  [node]
 5: 0x1119788 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 6: 0x10ef8a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 7: 0x10f0a35 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0x10cd156 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node]
 9: 0x10bed84 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node]
10: 0x10c2097 v8::internal::FactoryBase<v8::internal::Factory>::NewRawTwoByteString(int, v8::internal::AllocationType) [node]
11: 0x1232e67 v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [node]
12: 0x1238d4d  [node]
13: 0x123acb9 v8::internal::JsonParser<unsigned short>::ParseJson(v8::internal::Handle<v8::internal::Object>) [node]
14: 0xf8175e v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
15: 0x1963df6  [node]
[1]    91884 IOT instruction (core dumped)  codeburn

Root cause

PR #67 introduced readViaStream in src/fs-utils.ts as the "memory-safe" path for files ≥ 8 MB, but the implementation defeats itself:

async function readViaStream(filePath: string): Promise<string> {
  const chunks: string[] = []
  const stream = createReadStream(filePath, { encoding: 'utf-8' })
  const rl = createInterface({ input: stream, crlfDelay: Infinity })
  for await (const line of rl) chunks.push(line)
  return chunks.join('\n')  // ← full file reconstructed in memory
}

Every line is pushed into chunks[], then chunks.join('\n') reassembles the complete file as a single string — the same peak allocation as a plain readFile. The stream only operates at the I/O layer.

Callers then compound it with a second full copy:

const content = await readSessionFile(filePath)   // full string
const lines = content.split('\n')                 // second full copy

With FILE_READ_CONCURRENCY = 16 and files up to 128 MB, theoretical peak is 16 × 128 MB × ~3 = ~6 GB — right where the crash lands.

Fix

The fix already exists in the same file: readSessionLines is a proper async generator that yields one line at a time and never holds the full file in memory. It just isn't used by the two hot-path callers.

Switching scanJsonlFile (src/optimize.ts) and parseSessionFile (src/parser.ts) to iterate readSessionLines directly — instead of readSessionFile + split('\n') — eliminates the full-string allocation entirely. No concurrency reduction needed: with true line-by-line streaming, 16 concurrent files each hold only one line at a time.

I have a local fix ready with two new tests (a spy test confirming readSessionLines is called and readSessionFile is not, plus a 500-entry correctness test). Happy to open a PR if this diagnosis looks right to you.