Neo4j transaction too big

[georgesafta]

Describe the bug
The analyzer creates a huge Neo4J transaction which may conflict with the Neo4J configuration.

To Reproduce
Steps to reproduce the behavior:

1. Ran on superset (frontend)
   bun run start -- -i ../superset/superset-frontend/ --app-name superset-frontend --emit neo4j --neo4j-uri bolt://localhost:7687 --neo4j-user neo4j --neo4j-password somepassword

Expected behavior
The graph should be ingested without any issues.

Logs

[codeanalyzer-ts] FATAL Neo4jError: The allocation of an extra 2.0 MiB would use more than the limit 2.7 GiB. Currently using 2.7 GiB. dbms.memory.transaction.total.max threshold reached

    at captureStacktrace (/root/projects/python/codeanalyzer-typescript/node_modules/neo4j-driver-core/lib/result.js:624:21)
    at new Result (/root/projects/python/codeanalyzer-typescript/node_modules/neo4j-driver-core/lib/result.js:112:23)
    at <anonymous> (/root/projects/python/codeanalyzer-typescript/node_modules/neo4j-driver-core/lib/session.js:224:29)
    at <anonymous> (/root/projects/python/codeanalyzer-typescript/node_modules/neo4j-driver-core/lib/session.js:188:27)
    at <anonymous> (/root/projects/python/codeanalyzer-typescript/src/build/neo4j/bolt.ts:120:31)
    at withSession (/root/projects/python/codeanalyzer-typescript/src/build/neo4j/bolt.ts:249:18)
    at boltWriter (/root/projects/python/codeanalyzer-typescript/src/build/neo4j/bolt.ts:118:13)
    at async emitNeo4j (/root/projects/python/codeanalyzer-typescript/src/utils/serialize.ts:51:11)
    at async emit (/root/projects/python/codeanalyzer-typescript/src/utils/serialize.ts:17:11)
    at async main (unknown)
    at processTicksAndRejections (native:7:39)

Additional context
No

[rahlk]

Looked into this. Short version: the crash is real, but it is probably a symptom of something worse, and the crash may well have saved you.

What the analyzer does before it writes anything

When it connects to Neo4j, it first checks what schema version the database was written with. If that does not match its own, it assumes the database holds leftovers from an old version of this analyzer and clears them out before writing. That cleanup is the first thing it runs, and it is the line your stack trace points at (bolt.ts:120).

The cleanup statement is:

MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode DETACH DELETE n

In English: delete every node that belongs to a source file but is not labelled :CanNode. :CanNode is a label only the current TypeScript analyzer applies, so anything without it was assumed to be old TypeScript output.

Why that assumption is wrong

codeanalyzer-python and codeanalyzer-java both tag their nodes with _module, and neither of them uses :CanNode — I checked:

• python: _module is on four node types (codeanalyzer/neo4j/schema.py:91,107,131,160); CanNode appears nowhere in the repo
• java: _module is set in GraphProjector.java:256,273; CanNode appears in zero files

So that predicate is not a description of "old TypeScript nodes". It is an exact description of every python and java node in the database. If you pointed this at a Neo4j that already held a python or java graph, the TypeScript analyzer tried to delete all of it before writing anything of its own.

Why the error is the good outcome

The delete ran as one big transaction. It hit the memory ceiling, failed, and rolled back — so nothing was actually deleted. Had the foreign graph been smaller, or the server given more memory, the delete would have succeeded silently and you would have lost the python/java graph with no error at all.

So the memory error is not really the bug. The bug is that the cleanup is not scoped to this analyzer's own nodes.

Is that what happened to you?

I could not reproduce the crash from a TypeScript-only database. I ran superset-frontend into a clean Neo4j capped at the same 2.7 GiB you hit (124,866 nodes / 314,675 edges): it succeeded, twice, in about a minute. I also tried the cleanup against 200,000 and then 300,000 synthetic leftover nodes with relationships — Neo4j deleted them without complaint. Nothing blew up until I looked at what a shared database would contain.

To confirm, could you run this against the database you pushed to and say what it returns?

MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode RETURN count(n)

That is exactly the set the failing statement tried to delete. A large number, especially with python or java nodes in that database, confirms it. Zero would mean I am looking in the wrong place.

Also useful: which commit you were on (your line 120 matches v1.1.0), and whether that Neo4j had a python or java graph in it.

What needs fixing

1. Scope the cleanup to TypeScript-owned nodes. The old 1.x nodes it is meant to catch carried twin labels (:Module:TSModule, :Symbol:TSCallable), so it can identify them without matching anything a sibling analyzer wrote.
2. Batch it, so a large delete cannot exhaust transaction memory. This bounds the damage but does not fix the targeting — both are needed.
3. The third cleanup statement has the same problem in a different form: MATCH (a:Application) WHERE a.id IS NULL DETACH DELETE a. :Application is a deliberately shared, unprefixed label, so that can delete another analyzer's application node.

Unrelated thing found while digging

content_hash is never actually written to the graph — all 1,841 module nodes in my test database have it as NULL. The projection strips it (project.ts:231 even says so), but the incremental push still reads it to decide which modules changed. It always reads nothing, so it always concludes everything changed, so every push re-uploads the whole graph. That is not your crash, but on a memory-constrained server it guarantees every push does the maximum possible work. Worth its own issue.