fix(neo4j): stream the Cypher snapshot instead of building it as one String - #211
Merged
Conversation
…String
--emit neo4j died on a large repository before writing a byte:
OutOfMemoryError: UTF16 String size is 1169246895, should be less than 1073741823
at CypherWriter.renderCypher(CypherWriter.java:82)
Not heap exhaustion, so no -Xmx avoided it: renderCypher collected every
statement into a list and collapsed it with String.join, and the result has to
fit in one array. A script carrying any non-Latin-1 character is stored as
UTF16, which halves the ceiling to ~1.07 billion characters. ThingsBoard v4.0
renders 1.169 billion, so the join threw and left a zero-byte graph.cypher.
CypherWriter now streams: writeCypher(Appendable, rows, appName) emits each
statement as it is produced, so nothing larger than one 500-row batch is ever a
String and peak memory no longer scales with the graph. renderCypher stays as a
thin wrapper over it for tests and small callers, documented as unsuitable for
user-facing output, so the two cannot drift.
Statements are separated by newlines rather than terminated by them, matching
String.join exactly. Getting this wrong appends one byte, which a test comparing
renderCypher against writeCypher cannot see now that the former delegates to the
latter -- so the boundary is pinned directly, and the whole script was diffed
against the v3.0.0 release build.
Neo4jEmitter writes through Files.newBufferedWriter in explicit UTF-8. It used a
bare FileWriter, which encodes in the platform default charset; on Java 11 that
is not UTF-8 everywhere, so non-ASCII could already be mangled.
Verified on ThingsBoard v4.0 (4131 files): the v3.0.0 jar exits 1 with the OOM
above and writes 0 bytes; this build exits 0 in 5m20s and writes a 1.17 GB
script whose 4063 UNWIND blocks are all balanced. On daytrader8 the emitted
graph.cypher is byte-identical to the release.
Closes #209
georgesafta
approved these changes
Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
--emit neo4jdied on ThingsBoard v4.0 (4131 Java files) before writing a byte, and left a zero-bytegraph.cypherbehind:(Reported in the field as
Requested string length exceeds VM limitat the same line — same defect, different construction path.)It looks like
renderCyphercollected every statement into aList<String>and collapsed it withString.join, and that result has to fit in one array. A script containing any non-Latin-1 character is stored as UTF16, halving the ceiling to ~1.07 billion chars. ThingsBoard renders 1.169 billion. No-Xmxvalue avoids this.Fix
CypherWriter.writeCypher(Appendable, rows, appName)streams each statement as it is produced. Nothing larger than one 500-row batch is ever aString, so peak memory is bounded by the batch rather than by the graph.renderCypherremains as a thin wrapper over it — kept for tests and small callers, documented as unsuitable for user-facing output, and delegating so the two cannot drift.Neo4jEmitterwrites throughFiles.newBufferedWriterin explicit UTF-8. It previously used a bareFileWriter, which encodes in the platform default charset; on Java 11 that is not UTF-8 everywhere, so non-ASCII in identifiers or captured source could already be mangled. Same line, distinct defect — called out rather than folded in silently.The one subtlety
Statements are separated by newlines, not terminated by them, matching
String.joinexactly. Getting this wrong appends a single trailing byte — and a test comparingrenderCyphertowriteCyphercannot catch it, because the former now delegates to the latter and so compares the implementation with itself. I hit exactly this: the first implementation produced 33172816 bytes against the release's 33172815.So the boundary is pinned directly by
statementsAreSeparatedByNewlinesNotTerminatedByThem, and the full script was diffed against the v3.0.0 release build.Verification
On ThingsBoard v4.0, same command, same input:
OutOfMemoryError)graph.cypherThe emitted script is well-formed: 4063
UNWINDblocks, all balanced against their] AS rowclosers, 20 constraints, correct trailing newline.On
daytrader8the emittedgraph.cypheris byte-identical to the one the v3.0.0 release produces — 33172815 bytes,cmpclean.Suite: 491 tests, 1 failure, 4 skips — the failure is the pre-existing Docker-dependent
CodeAnalyzerIntegrationTest > initializationError, unrelated.Closes #209