Skip to content

chore(core): fix WalPurgeJob error on concurrent table drop - #6898

Merged
bluestreak01 merged 5 commits into
masterfrom
ia_fix_flaky_wal_purge
Mar 30, 2026
Merged

chore(core): fix WalPurgeJob error on concurrent table drop#6898
bluestreak01 merged 5 commits into
masterfrom
ia_fix_flaky_wal_purge

Conversation

@glasstiger

@glasstiger glasstiger commented Mar 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • WalPurgeJob.fetchSequencerPairs() catches Throwable (not just CairoException) when
    getTableMetadata() fails for a concurrently dropped table. A concurrent drop can close
    the metadata pool tenant's txFile during refresh, causing NullPointerException rather
    than CairoException.
  • AbstractMultiTenantPool.get0() catches Throwable (not just CairoException) in the
    newTenant path, so the pool slot is always released on failure.

Test plan

  • New test testGetTableMetadataNpeForDroppedTable simulates the race by intercepting
    FilesFacade.openRO to drop the table and throw NPE during metadata tenant creation
    inside the purge job. Verified: fails without the fix, passes with it.
  • Full WalPurgeJobTest suite passes (22 tests).

An instance of the error we are fixing:

2026-03-24T00:13:17.6906336Z 2026-03-24T00:13:17.689767Z I i.q.c.w.s.TableSequencerAPI dropping wal table [table=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.6908209Z 2026-03-24T00:13:17.689993Z I qdb_ent::wal::uploader L1234 drained queue [t=testTableDroppedWithSegments~11, events=[R9/seg_close/1/0, R10/wal_close/1]]
2026-03-24T00:13:17.6909415Z 2026-03-24T00:13:17.690001Z I qdb_ent::wal::uploader L1583 no new transactions to upload [r=R10, t=testTableDroppedWithSegments~11, last_uploaded_txn=Some(1)]
2026-03-24T00:13:17.6910145Z 2026-03-24T00:13:17.690392Z I qdb_ent::wal::uploader L1909 cleared segment pending file [r=R9, t=testTableDroppedWithSegments~11, wal_id=1, segment_id=0]
2026-03-24T00:13:17.6911319Z 2026-03-24T00:13:17.690400Z I qdb_ent::wal::uploader L1931 wal fully closed [r=R10, t=testTableDroppedWithSegments~11, wal_id=1]
2026-03-24T00:13:17.6914150Z 2026-03-24T00:13:17.690413Z I qdb_ent::wal::uploader L1294 draining queue [t=testTableDroppedWithSegments~11, last_notified_txn=Some(1), last_uploaded_txn=Some(1), window=false, block=true]
2026-03-24T00:13:17.6927930Z 2026-03-24T00:13:17.691411Z I i.q.c.w.s.TableSequencerImpl closed table sequencer [table=testTableDroppedWithSegments~11, distressed=false, dropped=true]
2026-03-24T00:13:17.6935474Z 2026-03-24T00:13:17.691493Z I i.q.c.w.WalWriter open [table=sys.acl_permissions~3]
2026-03-24T00:13:17.6937927Z 2026-03-24T00:13:17.691868Z I qdb_ent::wal::uploader L1234 drained queue [t=testTableDroppedWithSegments~11, events=[R11/dropped/2]]
2026-03-24T00:13:17.6947025Z 2026-03-24T00:13:17.691966Z I qdb_ent::wal::uploader L1481 reopened txn log [r=R11, t=testTableDroppedWithSegments~11, last_notified_txn=2, new reader.max_txn()=2]
2026-03-24T00:13:17.6959606Z 2026-03-24T00:13:17.691973Z I qdb_ent::wal::uploader L1611 advancing txns [r=R11, t=testTableDroppedWithSegments~11, from_txn=2, to_txn=2, mode=Full]
2026-03-24T00:13:17.6968720Z 2026-03-24T00:13:17.691988Z I qdb_ent::wal::uploader L1726 uploads spawned [t=testTableDroppedWithSegments~11, r=R11, txn=2, table_status=Dropped, up_meta=Some(2), total_uploads=2, segments=[]]
2026-03-24T00:13:17.6970755Z 2026-03-24T00:13:17.691998Z I qdb_ent::wal::uploader L2158 uploading sequencer meta [r=R11, upload_index=U0, t=testTableDroppedWithSegments~11, last_txn=2]
2026-03-24T00:13:17.6972037Z 2026-03-24T00:13:17.693936Z I c.q.c.w.t.UploadWalDirectoryPolicy created upload.pending file [path=C:\Users\VSSADM~1\AppData\Local\Temp\junit3804217777797318055\dbRoot\db\sys.acl_permissions~3\wal1\0\.pending\upload.pending]
2026-03-24T00:13:17.6973144Z 2026-03-24T00:13:17.695308Z I qdb_ent::wal::uploader L571 uploading [r=R11, u=U1, size=56/64=0.88x, key=testTableDroppedWithSegments~11/_txn_parts/0.zst]
2026-03-24T00:13:17.6973958Z 2026-03-24T00:13:17.695368Z I qdb_ent::wal::uploader L571 uploading [r=R11, u=U0, size=131400/233=563.95x, key=testTableDroppedWithSegments~11/txn_seq.meta.tar.zst]
2026-03-24T00:13:17.7015175Z 2026-03-24T00:13:17.701306Z I i.q.c.w.WalWriter opened WAL segment [path=\sys.acl_permissions~3\wal1\0]
2026-03-24T00:13:17.7019200Z 2026-03-24T00:13:17.701601Z I i.q.c.TableWriter closing table files [table=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.7054917Z 2026-03-24T00:13:17.705276Z I i.q.c.p.WriterPool closed [table=testTableDroppedWithSegments~11, reason=DISTRESSED, by=9231]
2026-03-24T00:13:17.7073287Z 2026-03-24T00:13:17.707028Z I qdb_ent::wal::uploader L593 uploaded [r=R11, u=U1, throughput=0.01 MB/s]
2026-03-24T00:13:17.7248616Z 2026-03-24T00:13:17.713277Z I i.q.c.w.WalWriter closed [table=sys.acl_permissions~3]
2026-03-24T00:13:17.7250217Z 2026-03-24T00:13:17.713604Z I i.q.c.p.TableMetadataPool shutting down, table is left behind [table=sys.acl_permissions~3]
2026-03-24T00:13:17.7251148Z 2026-03-24T00:13:17.714515Z I server-main QuestDB is shutting down...
2026-03-24T00:13:17.7252493Z 2026-03-24T00:13:17.715486Z I qdb_ent::wal::uploader L593 uploaded [r=R11, u=U0, throughput=0.01 MB/s]
2026-03-24T00:13:17.7253589Z 2026-03-24T00:13:17.715583Z I qdb_ent::wal::uploader L1758 sequencer and segment uploads complete [t=testTableDroppedWithSegments~11, r=R11]
2026-03-24T00:13:17.7254785Z 2026-03-24T00:13:17.721778Z I i.q.c.w.WalPurgeJob table=testTableDroppedWithSegments~11, discovered=[(wal1),(1,0)]
2026-03-24T00:13:17.7264054Z 2026-03-24T00:13:17.725644Z I i.q.s.FilesFacadeImpl completed rmdir with subdirectories [path=C:\Users\VSSADM~1\AppData\Local\Temp\junit3804217777797318055\dbRoot\db\testTableDroppedWithSegments~11\wal1, subDirDepth=2]
2026-03-24T00:13:17.7265404Z 2026-03-24T00:13:17.725662Z I i.q.c.w.WalPurgeJob table is dropped, but has WALs containing segments with pending tasks [table=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.7309432Z 2026-03-24T00:13:17.729565Z I qdb_ent::wal::uploader::indexer L591 uploaded index.msgpack [r=R11, sync_id=16124208418619802935]
2026-03-24T00:13:17.7349320Z 2026-03-24T00:13:17.732544Z I qdb_ent::wal::uploader L2142 table sequencer ready for deletion, removed upload.pending marker file [r=R11, path=C:\Users\VSSADM~1\AppData\Local\Temp\junit3804217777797318055\dbRoot\db\testTableDroppedWithSegments~11\txn_seq\.pending\upload.pending]
2026-03-24T00:13:17.7350590Z 2026-03-24T00:13:17.732552Z I qdb_ent::wal::uploader L1800 table updated [r=R11, t=testTableDroppedWithSegments~11, txn=2]
2026-03-24T00:13:17.7351540Z 2026-03-24T00:13:17.732555Z I qdb_ent::wal::uploader L1860 table dropped: exiting process_events_task soon [r=R11, t=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.7352582Z 2026-03-24T00:13:17.732558Z I qdb_ent::wal::uploader L1981 moribund, table is dropped [t=testTableDroppedWithSegments~11, closed_segments_queue=0, open_segments=[]]
2026-03-24T00:13:17.7353352Z 2026-03-24T00:13:17.732624Z I qdb_ent::wal::uploader L2002 cleaning up after completing table task [t=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.7354121Z 2026-03-24T00:13:17.732631Z I qdb_ent::wal::uploader L2052 notifying uploads complete [t=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.7409760Z 2026-03-24T00:13:17.740732Z I qdb_ent::wal::downloader L703 processing table on poll event [poll_id=P65, sync_id=16124208418619802935, r=R17, t=testTableDroppedWithSegments~11, last_txn=2]
2026-03-24T00:13:17.7410905Z 2026-03-24T00:13:17.740740Z I qdb_ent::wal::downloader L737 table has been dropped [r=R17, t=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.7449209Z 2026-03-24T00:13:17.743790Z I i.q.c.MetadataCache dropped [table=testTableDroppedWithSegments~11]
2026-03-24T00:13:17.7450378Z 2026-03-24T00:13:17.744127Z I qdb_ent::wal::downloader L784 updated last txn for dropped table [r=R17, t=testTableDroppedWithSegments~11, last_txn=2]
2026-03-24T00:13:17.7633396Z 2026-03-24T00:13:17.762450Z C server-main unhandled error [job=io.questdb.cairo.wal.WalPurgeJob@43d70c1b, ex=java.lang.NullPointerException: Cannot invoke "io.questdb.cairo.TxReader.getTxn()" because "this.txFile" is null
2026-03-24T00:13:17.7634639Z 	at io.questdb.cairo.pool.TableReaderMetadataTenantImpl.acquireTxn(TableReaderMetadataTenantImpl.java:160)
2026-03-24T00:13:17.7635403Z 	at io.questdb.cairo.pool.TableReaderMetadataTenantImpl.reload(TableReaderMetadataTenantImpl.java:139)
2026-03-24T00:13:17.7636219Z 	at io.questdb.cairo.pool.TableReaderMetadataTenantImpl.refresh(TableReaderMetadataTenantImpl.java:132)
2026-03-24T00:13:17.7637300Z 	at io.questdb.cairo.pool.AbstractMultiTenantPool.get0(AbstractMultiTenantPool.java:316)
2026-03-24T00:13:17.7637996Z 	at io.questdb.cairo.pool.AbstractMultiTenantPool.get(AbstractMultiTenantPool.java:83)
2026-03-24T00:13:17.7638691Z 	at io.questdb.cairo.CairoEngine.getTableMetadata(CairoEngine.java:1062)
2026-03-24T00:13:17.7639331Z 	at io.questdb.cairo.CairoEngine.getTableMetadata(CairoEngine.java:1040)
2026-03-24T00:13:17.7639957Z 	at io.questdb.cairo.wal.WalPurgeJob.fetchSequencerPairs(WalPurgeJob.java:402)
2026-03-24T00:13:17.7640573Z 	at io.questdb.cairo.wal.WalPurgeJob.broadSweep(WalPurgeJob.java:195)
2026-03-24T00:13:17.7641212Z 	at io.questdb.cairo.wal.seq.TableSequencerAPI.forAllWalTables(TableSequencerAPI.java:166)
2026-03-24T00:13:17.7642599Z 	at io.questdb.cairo.wal.WalPurgeJob.broadSweep(WalPurgeJob.java:174)
2026-03-24T00:13:17.7643218Z 	at io.questdb.cairo.wal.WalPurgeJob.runSerially(WalPurgeJob.java:552)
2026-03-24T00:13:17.7643827Z 	at io.questdb.mp.SynchronizedJob.run(SynchronizedJob.java:40)
2026-03-24T00:13:17.7644399Z 	at io.questdb.mp.Worker.run(Worker.java:152)
2026-03-24T00:13:17.7644875Z ]

@coderabbitai

coderabbitai Bot commented Mar 24, 2026 •
edited
Loading

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d0f438ea-c5d5-48cc-bf4d-f3fc193f239b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Walkthrough

The pull request broadens exception handling in tenant pool creation and WAL purge logic to catch Throwable instead of just CairoException, enabling graceful handling of NullPointerException during concurrent table drops. A regression test validates this behavior.

Changes

Cohort / File(s) Summary
Exception handling generalization
core/src/main/java/io/questdb/cairo/pool/AbstractMultiTenantPool.java, core/src/main/java/io/questdb/cairo/wal/WalPurgeJob.java		 Broadened catch blocks from CairoException to Throwable to handle broader error types (including NullPointerException) during concurrent table operations, with slot cleanup and rethrow on error.
Test coverage and modernization
core/src/test/java/io/questdb/test/cairo/wal/WalPurgeJobTest.java		 Added regression test testGetTableMetadataNpeForDroppedTable with controlled race condition simulating NullPointerException during metadata reads; updated DeletionEvent.equals() to use Java pattern matching and added static import for TableUtils.META_FILE_NAME.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Suggested labels

Bug, storage

Suggested reviewers

  • bluestreak01
  • ideoma
🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the main change: fixing a WalPurgeJob error that occurs during concurrent table drops.
Description check ✅ Passed The description provides a clear explanation of the problem (NullPointerException during concurrent drops), the solution (catching Throwable instead of CairoException), and a test plan with regression test details.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ia_fix_flaky_wal_purge

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@glasstiger glasstiger changed the title chore(core): fix WalPurgeJob - dropTable() race fix(core): fix WalPurgeJob crash on concurrent table drop Mar 24, 2026
@glasstiger glasstiger changed the title fix(core): fix WalPurgeJob crash on concurrent table drop chore(core): fix WalPurgeJob error on concurrent table drop Mar 24, 2026
@glasstiger

glasstiger commented Mar 24, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Mar 24, 2026

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

bluestreak01
bluestreak01 requested changes Mar 28, 2026
View reviewed changes
Comment thread core/src/test/java/io/questdb/test/cairo/wal/WalPurgeJobTest.java Outdated
bluestreak01 and others added 2 commits March 27, 2026 22:45
@bluestreak01 @claude
narrow catch to CairoException | NullPointerException
46b5ef4 
catch(Throwable) risked silently swallowing Error types
(e.g. OutOfMemoryError) when a table happened to be dropped.
Narrow to the two exception types the concurrent-drop race
actually produces: CairoException (normal path) and
NullPointerException (when txFile is nulled mid-refresh).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bluestreak01
Merge branch 'master' into ia_fix_flaky_wal_purge
f7fe609
@glasstiger

glasstiger commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

[PR Coverage check]

😍 pass : 3 / 3 (100.00%)

file detail

path covered line new line coverage
🔵 io/questdb/cairo/wal/WalPurgeJob.java 1 1 100.00%
🔵 io/questdb/cairo/pool/AbstractMultiTenantPool.java 2 2 100.00%

@bluestreak01
bluestreak01 merged commit adb1505 into master Mar 30, 2026
51 checks passed
@bluestreak01
bluestreak01 deleted the ia_fix_flaky_wal_purge branch March 30, 2026 20:20
@coderabbitai coderabbitai Bot mentioned this pull request Apr 30, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bluestreak01 bluestreak01 bluestreak01 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@glasstiger @bluestreak01