Skip to content

fix(neo4j): stream the Cypher snapshot instead of building it as one String - #211

Merged
rahlk merged 1 commit into
mainfrom
fix/issue-209-stream-cypher
Sep 1, 2026
Merged

fix(neo4j): stream the Cypher snapshot instead of building it as one String#211
rahlk merged 1 commit into
mainfrom
fix/issue-209-stream-cypher

Conversation

@rahlk

@rahlk rahlk commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Problem

--emit neo4j died on ThingsBoard v4.0 (4131 Java files) before writing a byte, and left a zero-byte graph.cypher behind:

OutOfMemoryError: UTF16 String size is 1169246895, should be less than 1073741823
    at CypherWriter.renderCypher(CypherWriter.java:82)

(Reported in the field as Requested string length exceeds VM limit at the same line — same defect, different construction path.)

It looks like renderCypher collected every statement into a List<String> and collapsed it with String.join, and that result has to fit in one array. A script containing any non-Latin-1 character is stored as UTF16, halving the ceiling to ~1.07 billion chars. ThingsBoard renders 1.169 billion. No -Xmx value avoids this.

Fix

CypherWriter.writeCypher(Appendable, rows, appName) streams each statement as it is produced. Nothing larger than one 500-row batch is ever a String, so peak memory is bounded by the batch rather than by the graph. renderCypher remains as a thin wrapper over it — kept for tests and small callers, documented as unsuitable for user-facing output, and delegating so the two cannot drift.

Neo4jEmitter writes through Files.newBufferedWriter in explicit UTF-8. It previously used a bare FileWriter, which encodes in the platform default charset; on Java 11 that is not UTF-8 everywhere, so non-ASCII in identifiers or captured source could already be mangled. Same line, distinct defect — called out rather than folded in silently.

The one subtlety

Statements are separated by newlines, not terminated by them, matching String.join exactly. Getting this wrong appends a single trailing byte — and a test comparing renderCypher to writeCypher cannot catch it, because the former now delegates to the latter and so compares the implementation with itself. I hit exactly this: the first implementation produced 33172816 bytes against the release's 33172815.

So the boundary is pinned directly by statementsAreSeparatedByNewlinesNotTerminatedByThem, and the full script was diffed against the v3.0.0 release build.

Verification

On ThingsBoard v4.0, same command, same input:

v3.0.0 release jar this build
exit 1 (OutOfMemoryError) 0
graph.cypher 0 bytes 1.17 GB
wall clock 5m20s

The emitted script is well-formed: 4063 UNWIND blocks, all balanced against their ] AS row closers, 20 constraints, correct trailing newline.

On daytrader8 the emitted graph.cypher is byte-identical to the one the v3.0.0 release produces — 33172815 bytes, cmp clean.

Suite: 491 tests, 1 failure, 4 skips — the failure is the pre-existing Docker-dependent CodeAnalyzerIntegrationTest > initializationError, unrelated.

Closes #209

…String

--emit neo4j died on a large repository before writing a byte:

    OutOfMemoryError: UTF16 String size is 1169246895, should be less than 1073741823
        at CypherWriter.renderCypher(CypherWriter.java:82)

Not heap exhaustion, so no -Xmx avoided it: renderCypher collected every
statement into a list and collapsed it with String.join, and the result has to
fit in one array. A script carrying any non-Latin-1 character is stored as
UTF16, which halves the ceiling to ~1.07 billion characters. ThingsBoard v4.0
renders 1.169 billion, so the join threw and left a zero-byte graph.cypher.

CypherWriter now streams: writeCypher(Appendable, rows, appName) emits each
statement as it is produced, so nothing larger than one 500-row batch is ever a
String and peak memory no longer scales with the graph. renderCypher stays as a
thin wrapper over it for tests and small callers, documented as unsuitable for
user-facing output, so the two cannot drift.

Statements are separated by newlines rather than terminated by them, matching
String.join exactly. Getting this wrong appends one byte, which a test comparing
renderCypher against writeCypher cannot see now that the former delegates to the
latter -- so the boundary is pinned directly, and the whole script was diffed
against the v3.0.0 release build.

Neo4jEmitter writes through Files.newBufferedWriter in explicit UTF-8. It used a
bare FileWriter, which encodes in the platform default charset; on Java 11 that
is not UTF-8 everywhere, so non-ASCII could already be mangled.

Verified on ThingsBoard v4.0 (4131 files): the v3.0.0 jar exits 1 with the OOM
above and writes 0 bytes; this build exits 0 in 5m20s and writes a 1.17 GB
script whose 4063 UNWIND blocks are all balanced. On daytrader8 the emitted
graph.cypher is byte-identical to the release.

Closes #209
@rahlk rahlk self-assigned this Aug 31, 2026
@rahlk
rahlk requested a review from georgesafta August 31, 2026 20:39
@rahlk
rahlk merged commit b241dcd into main Sep 1, 2026
@rahlk
rahlk deleted the fix/issue-209-stream-cypher branch September 1, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(neo4j): --emit neo4j OOMs on a large repository — the whole Cypher script is built as one String

2 participants