Import Database Neo4j - OutOfMemoryError

[RobertoSannino]

Version: 26.3.2 and 26.4.1-SNAPSHOT
Environment: Docker

While trying to import a big dataset (200+GB) exported from Neo4j using CALL apoc.export.json.all("all_20260211.json",{useTypes:true});
I incur in OutOfMemory errors even using commitEvery = 5, it seems like it tries to load all the json lines for a specific Vertex type all in once, without properly batching them.

The command I am executing from the command line is the following:
{test}> IMPORT DATABASE file:///home/arcadedb/data/import/test_all_20260401.json with commitEvery = 500, forceDatabaseCreate = true, wal = false;

Here the output:

...
Parsed 0 (0/sec 0%) 0 records (0/sec) 0 vertices (0/sec) 0 edges (0/sec 0 skipped) 0 linked edges (0/sec 0%) updated documents 0 (0%)
Parsed 0 (0/sec 0%) 0 records (0/sec) 0 vertices (0/sec) 0 edges (0/sec 0 skipped) 0 linked edges (0/sec 0%) updated documents 0 (0%)
Parsed 0 (0/sec 0%) 0 records (0/sec) 0 vertices (0/sec) 0 edges (0/sec 0 skipped) 0 linked edges (0/sec 0%) updated documents 0 (0%)
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid409.hprof ...
Heap dump file created [34501946522 bytes in 53.464 secs]

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ArcadeDB TransactionManager test"
Parsed 0 (0/sec 0%) 0 records (0/sec) 0 vertices (0/sec) 0 edges (0/sec 0 skipped) 0 linked edges (0/sec 0%) updated documents 0 (0%)

ERROR:
com.arcadedb.exception.CommandExecutionException: Error on importing database
	at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:77)
	at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:92)
	at com.arcadedb.query.sql.parser.SimpleExecStatement.execute(SimpleExecStatement.java:72)
	at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:73)
	at com.arcadedb.query.sql.SQLQueryEngine.command(SQLQueryEngine.java:113)
	at com.arcadedb.query.QueryEngine.command(QueryEngine.java:90)
	at com.arcadedb.database.LocalDatabase.command(LocalDatabase.java:1492)
	at com.arcadedb.console.Console.executeSQL(Console.java:632)
	at com.arcadedb.console.Console.execute(Console.java:293)
	at com.arcadedb.console.Console.parse(Console.java:787)
	at com.arcadedb.console.Console.interactiveMode(Console.java:149)
	at com.arcadedb.console.Console.execute(Console.java:206)
	at com.arcadedb.console.Console.main(Console.java:166)
Caused by: java.lang.OutOfMemoryError: Java heap space

[robfrank]

which memory settings are you using to run the arcadeDB container?

[RobertoSannino]

  arcadedb:
    container_name: arcadedb
    image: arcadedata/arcadedb:26.3.2
    environment:
      ARCADEDB_OPTS_MEMORY: -Xms1G -Xmx32G
      JAVA_OPTS: >
        -Xms800M -Xmx32G
        ...

[lvca]

I think it's time to push on main the nee optimized Neo4j importer that uses the same underlying architecture of the GraphImporter: https://arcadedb.com/blog/graphbatch-up-to-8x-faster-graph-ingestion/. Working on merging it

[lvca]

It has been merged on main, please @RobertoSannino give it a try after built a new distro from main source (mvn -T 1C clean install -DskipTests -> 26.4.1-SNAPSHOT). Same API, same way, but the new version leverages the new GraphBatch API to be much faster with less RAM used

[lvca]

@RobertoSannino also, if your neo4j database has numeric ids (always exported as string, but by default it's a number unless you use your own ids) the new importer uses >80% less RAM -> #3770

[RobertoSannino]

Thank you! I will give it a try asap

[lvca]

Closing for now because of no response. In case the issue persists, please comment this issue.