Problem
--emit neo4j fails on a large real-world repository (ThingsBoard v4.0, 4131 Java files):
Exception in thread "main" java.lang.OutOfMemoryError: Requested string length exceeds VM limit
at java.base/java.lang.String.join(String.java:3267)
at java.base/java.lang.String.join(String.java:3335)
at com.ibm.cldk.neo4j.CypherWriter.renderCypher(CypherWriter.java:82)
at com.ibm.cldk.neo4j.Neo4jEmitter.write(Neo4jEmitter.java:104)
This is not heap exhaustion, and more heap will not fix it. Requested string length exceeds VM limit means a single String was asked to exceed the JVM's maximum array length (~2.1 billion chars). -Xmx64g fails identically.
The cause is that the entire script is materialized as one String before a single byte is written:
// CypherWriter.java:82
return String.join("\n", out);
// Neo4jEmitter.java:104
w.write(CypherWriter.renderCypher(rows, name));
Peak memory is therefore roughly three concurrent copies of the whole script — the List<String> of statements, the joined String, and the encoder buffer — and the joined String additionally has to fit in one array.
Why it has not been seen before
Script size scales with the repository. On the daytrader8 fixture (141 files) graph.cypher is 33 MB and nothing is noticeable. ThingsBoard is 4131 files, and its -a 1 run also produces 4772 artifacts and 42,768 config keys (see the sibling issue on config-key explosion, which inflates this further but is not the cause). The script passes the 2 GB ceiling and the join throws.
Scope boundary
Make the writer stream: emit each statement to the Writer as it is produced, so no full-script String ever exists and peak memory is one batch rather than the whole graph. Removes the ceiling rather than raising it.
Does not change a single byte of emitted Cypher — same statements, same order, same batching. Does not touch V2GraphProjector, the schema catalog, or BoltWriter (which already pushes per batch and does not have this failure mode).
Goals
Caveats and known risks
Neo4jEmitter writes through a bare FileWriter, which encodes in the platform default charset. On Java 11 that is not UTF-8 everywhere, so non-ASCII in identifiers or captured source can already be mangled today. Worth fixing in the same change since it is the same line, but it is a distinct defect — call it out separately rather than folding it in silently.
- Streaming without buffering would trade an OOM for a syscall per statement. Wrap in a
BufferedWriter.
- A test that renders the full fixture graph into memory to compare is fine at fixture scale; do not write a test that reproduces the 2 GB case.
Definition of done
Problem
--emit neo4jfails on a large real-world repository (ThingsBoard v4.0, 4131 Java files):This is not heap exhaustion, and more heap will not fix it.
Requested string length exceeds VM limitmeans a singleStringwas asked to exceed the JVM's maximum array length (~2.1 billion chars).-Xmx64gfails identically.The cause is that the entire script is materialized as one
Stringbefore a single byte is written:Peak memory is therefore roughly three concurrent copies of the whole script — the
List<String>of statements, the joinedString, and the encoder buffer — and the joinedStringadditionally has to fit in one array.Why it has not been seen before
Script size scales with the repository. On the
daytrader8fixture (141 files)graph.cypheris 33 MB and nothing is noticeable. ThingsBoard is 4131 files, and its-a 1run also produces 4772 artifacts and 42,768 config keys (see the sibling issue on config-key explosion, which inflates this further but is not the cause). The script passes the 2 GB ceiling and the join throws.Scope boundary
Make the writer stream: emit each statement to the
Writeras it is produced, so no full-scriptStringever exists and peak memory is one batch rather than the whole graph. Removes the ceiling rather than raising it.Does not change a single byte of emitted Cypher — same statements, same order, same batching. Does not touch
V2GraphProjector, the schema catalog, orBoltWriter(which already pushes per batch and does not have this failure mode).Goals
CypherWritergains a streaming entry point that writes to anAppendable/WriterincrementallyNeo4jEmitteruses it instead ofw.write(renderCypher(...))renderCypher(rows, appName)is kept as a thin wrapper over the streaming form, so existing callers and tests are unaffected and the two can never driftCaveats and known risks
Neo4jEmitterwrites through a bareFileWriter, which encodes in the platform default charset. On Java 11 that is not UTF-8 everywhere, so non-ASCII in identifiers or captured source can already be mangled today. Worth fixing in the same change since it is the same line, but it is a distinct defect — call it out separately rather than folding it in silently.BufferedWriter.Definition of done
--emit neo4jcompletes on ThingsBoard v4.0 and produces a loadablegraph.cypherCodeAnalyzerIntegrationTest