Skip to content

fix(neo4j): --emit neo4j OOMs on a large repository — the whole Cypher script is built as one String #209

Description

@rahlk

Problem

--emit neo4j fails on a large real-world repository (ThingsBoard v4.0, 4131 Java files):

Exception in thread "main" java.lang.OutOfMemoryError: Requested string length exceeds VM limit
        at java.base/java.lang.String.join(String.java:3267)
        at java.base/java.lang.String.join(String.java:3335)
        at com.ibm.cldk.neo4j.CypherWriter.renderCypher(CypherWriter.java:82)
        at com.ibm.cldk.neo4j.Neo4jEmitter.write(Neo4jEmitter.java:104)

This is not heap exhaustion, and more heap will not fix it. Requested string length exceeds VM limit means a single String was asked to exceed the JVM's maximum array length (~2.1 billion chars). -Xmx64g fails identically.

The cause is that the entire script is materialized as one String before a single byte is written:

// CypherWriter.java:82
return String.join("\n", out);

// Neo4jEmitter.java:104
w.write(CypherWriter.renderCypher(rows, name));

Peak memory is therefore roughly three concurrent copies of the whole script — the List<String> of statements, the joined String, and the encoder buffer — and the joined String additionally has to fit in one array.

Why it has not been seen before

Script size scales with the repository. On the daytrader8 fixture (141 files) graph.cypher is 33 MB and nothing is noticeable. ThingsBoard is 4131 files, and its -a 1 run also produces 4772 artifacts and 42,768 config keys (see the sibling issue on config-key explosion, which inflates this further but is not the cause). The script passes the 2 GB ceiling and the join throws.

Scope boundary

Make the writer stream: emit each statement to the Writer as it is produced, so no full-script String ever exists and peak memory is one batch rather than the whole graph. Removes the ceiling rather than raising it.

Does not change a single byte of emitted Cypher — same statements, same order, same batching. Does not touch V2GraphProjector, the schema catalog, or BoltWriter (which already pushes per batch and does not have this failure mode).

Goals

  • CypherWriter gains a streaming entry point that writes to an Appendable/Writer incrementally
  • Neo4jEmitter uses it instead of w.write(renderCypher(...))
  • renderCypher(rows, appName) is kept as a thin wrapper over the streaming form, so existing callers and tests are unaffected and the two can never drift
  • Byte-for-byte identical output to the current implementation on a fixture, proven by a test rather than asserted

Caveats and known risks

  • Neo4jEmitter writes through a bare FileWriter, which encodes in the platform default charset. On Java 11 that is not UTF-8 everywhere, so non-ASCII in identifiers or captured source can already be mangled today. Worth fixing in the same change since it is the same line, but it is a distinct defect — call it out separately rather than folding it in silently.
  • Streaming without buffering would trade an OOM for a syscall per statement. Wrap in a BufferedWriter.
  • A test that renders the full fixture graph into memory to compare is fine at fixture scale; do not write a test that reproduces the 2 GB case.

Definition of done

  • A test that fails before and passes after: the streamed output equals the previously-rendered string for a real fixture graph
  • --emit neo4j completes on ThingsBoard v4.0 and produces a loadable graph.cypher
  • Full suite green apart from the pre-existing Docker-dependent CodeAnalyzerIntegrationTest

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions