feat(python): expose Table.expire_snapshots - #967

zhuxiangyi · 2026-09-26T12:03:54Z

Purpose

Linked issue: #964 (part 3 of 4)

Stacked on #965. Only the last three commits, "feat(python): expose Table.expire_snapshots" and
its two tests, are new here.

Expose snapshot expiration from #965 to Python.

Brief change log

Table.expire_snapshots(retain_max=None, retain_min=None, older_than_ms=None, max_deletes=None) -> int in the Python binding. Unset arguments fall back to the table options, as in CALL sys.expire_snapshots. The call releases the GIL while it runs and returns the number of expired
snapshots.
Type stub and docstring in datafusion.pyi.

Tests

tests/test_table.py::test_expire_snapshots covers:
- the defaults keep recent snapshots;
- retain_max / retain_min;
- older_than_ms with max_deletes;
- the latest data and a tagged snapshot stay readable;
- invalid retention raises ValueError.
make install && make test for tests/test_table.py passes (24 tests), and so does clippy with
-D warnings.

API and Format

New Python method Table.expire_snapshots. No format change.

Documentation

Docstring in the type stub.

JingsongLi · 2026-10-01T16:03:56Z

Requirement fit: SUPPORTED. Python binding: CLEAN; shared expiration implementation: FINDINGS.

[P1] Preserve branch references before expiration (crates/paimon/src/table/expire_snapshots.rs:222-260, inherited from #965). With this exact Python extension, the existing branch_tables fixture reads branch row 1 successfully; main.expire_snapshots(retain_min=1, older_than_ms=2**62) expires snapshot 1, main still reads [1, 2], but the existing branch now raises ValueError/NotFound for its deleted manifest list. The branch snapshot/tag metadata is still live. Protect all live branch owners before deleting shared files, or reject unsupported expiration before mutation. The same underlying defect is described in #968 (comment).

[P2] Preserve persisted long-lived changelog owners (crates/paimon/src/table/expire_snapshots.rs:234-238 and snapshot_deletion.rs:324-326). Java writes changelog/changelog- immediately before removing snapshot-; interrupted expiration can legitimately leave both owners. Java's Changelog(Snapshot) copies changelogManifestList, and its incremental reader reads that manifest list. I reproduced this state using an actual input-changelog commit, persisted its snapshot JSON as changelog/changelog-1, committed snapshot 2, and called the new Python expiration method. The owner remains, but its previously existing changelog manifest list is deleted. Reject such unsupported lifecycle states before mutation or preserve their referenced files. Independent changelog retention policy can stay outside this PR's scope; leaving a persisted owner dangling is the safety issue.

Validation at 40eb555: built and installed the actual extension with maturin; all 24 existing Python table tests passed, including the new API's write/expire/read, tag, limit and invalid-retention coverage. Two additional real-filesystem Python probes above fail. The binding's GIL release, argument ownership and exception mapping were independently checked. Current main also conflicts in table/mod.rs; rebase and rerun integration before merge.

Additional shared-foundation review: #965 (comment) covers branch ownership, persisted long-lived changelog ownership, and an obsolete legacy Python DV left behind because expiry resolves only its bucket path. This PR inherits that deletion logic; the implementation differs only in the orphan cleaner helper visibility for #968.

zhuxiangyi · 2026-10-02T14:48:32Z

Thanks for building and probing the extension!

[P1] Branch references and [P2] persisted long-lived changelogs: both are fixed in the shared expiration code in #965 (e1d75d2). It now keeps everything that branches and long-lived changelogs reference, or fails before deleting anything. This branch is rebased onto it and onto current main.

I added your probe as a Python test in 13b76a2. test_expire_snapshots_keeps_branch_files uses the existing branch_tables fixture. After main.expire_snapshots(retain_min=1, older_than_ms=2**62), main still reads [1, 2] and the branch still reads [1]. make install && pytest tests/test_table.py passes (25 tests).

JingsongLi

Requirement fit: SUPPORTED. Python binding: CLEAN; shared expiration: FINDINGS. Built/installed the exact-head extension with maturin on Python 3.13. All 25 table tests, 4 fork/GIL tests, 27 Rust expiry tests and the persisted-changelog Python probe passed. The previous branch/changelog/legacy-DV fixes are verified. A real Python configured-directory expiry probe reproduces the shared P2 below. Current-main merge is clean; all 15 head CI checks are successful.

JingsongLi · 2026-10-03T09:50:37Z

+            Some(_) => BinaryRow::from_serialized_bytes(partition)?,
+        };
+        bucket_path(
+            &self.table_location,


[P2] Resolve expired data paths from the configured data root

With a real Python table configured data-file.path-directory=data, INSERT writes the old Parquet under <table>/data/bucket-0. After INSERT OVERWRITE, the new Python expire_snapshots(retain_min=1, retain_max=1) returns 1 and only snapshot 2 remains, but the expired snapshot's original Parquet still exists. Latest rows remain correct; physical reclamation silently fails because this helper attempts <table>/bucket-0 instead of the writer/read path. Use Table::data_file_location() for data and bucket-local sidecars/indexes, retaining table_location for metadata/global indexes, and add a configured-directory regression through the Python API. This is the same shared foundation defect reproduced on #968.

Fixed by the shared change in #965 (f7ba2d3): bucket directories now come from Table::data_file_location(). Following your probe, the Python regression test_expire_snapshots_with_data_directory creates a table with data-file.path-directory = data, inserts, overwrites, and calls expire_snapshots(retain_min=1, retain_max=1). It returns 1, only one file remains under data/bucket-0, and the table reads the new row.

JingsongLi

Additional finding from this review sweep: the expiration implementation at this reviewed head is byte-identical to #965. A new actual global-index/branch/overwrite failure probe on #965 reproduces the P2 below; an early-metadata-preflight control passes. This supplements the previous findings; it does not assert this exact probe was run through this PR's Python/automatic/orphan entry point.

JingsongLi · 2026-10-03T10:01:02Z

+            Ok(mut skipping) => {
+                skipping.extend(
+                    deletion
+                        .manifest_skipping_set(&external_snapshots, true)


[P2] Preflight all external owner metadata before deleting files

external_owners only validates base/delta data manifests up front. External index/changelog metadata are first checked here, after clean_data_files and changelog deletion. I built an actual global index, branched from its indexed snapshot, dropped the main index, overwrote main twice, then corrupted only the branch-owned index manifest. Expiration returns the Avro read error, leaves every snapshot JSON (including snapshot4), but has already deleted snapshot4's Parquet. This violates the new guarantee that an unreadable external owner aborts with nothing changed and leaves retained residual history unreadable after a failed call. A control that preflights manifest_skipping_set(external_snapshots,true) immediately after external_owners returns the same error with all physical data preserved and the probe passes. Build/reuse that skipping set before any data/changelog mutation, and cover optional index/changelog metadata failures.

Fixed in the shared expiration code of #965 (f7ba2d3). The external owners' skipping set, including their index and changelog manifests, is now built before any deletion, so an unreadable owner fails the run with nothing changed. The regression tests are described in the #965 thread.

zhuxiangyi · 2026-10-03T13:16:41Z

Round 2 is addressed; details are in the inline replies. The shared fixes are in #965 (f7ba2d3), and ad38d25 adds the Python regression for a configured data directory. The branch is rebased onto #965 and main c41ce20.

The ids in my previous reply predate two rebases: 13b76a2 is now c620c51, and e1d75d2 is now 74741ed.

make install && pytest tests/test_table.py passes (26 tests), and so does clippy with -D warnings.

JingsongLi

Requirement fit: SUPPORTED. Implementation: FINDINGS for production integration at ad38d25. The Python API itself has no new finding: I built and loaded this exact head extension, and all 26 existing table tests plus an additional argument/error/actual-expiry probe passed (27 total). All 31 core expiration tests also passed. Tags, branches, latest rows and configured data directories remain readable after expiration.

The automatic merge with current main bede665 fails the core test build with E0432 at the inherited snapshot-deletion import below. The shared foundation needs updating before this Python feature can merge or go to production.

JingsongLi · 2026-10-04T11:18:42Z

+    ManifestFileMeta, ManifestList, PartitionComputer, Snapshot,
+};
+use crate::table::index_file_path::{
+    committed_index_file_path, resolve_legacy_deletion_vector_entries,


[P1] Update snapshot deletion to the current main index-path API

The merge is textually clean, but this newly added module imports resolve_legacy_deletion_vector_entries, which current main no longer provides in table::index_file_path after #1016. On the automatic merge with bede665, the core test build fails with E0432: unresolved import crate::table::index_file_path::resolve_legacy_deletion_vector_entries. The passing head tests use the older foundation and do not establish that this PR can be merged now. Rebase/adapt the shared snapshot deletion code to current index-file resolution and rerun the merged core/SQL tests before production use.

Fixed in the shared code of #965 (5e9cbad). Index files are now resolved through committed_index_file_path only, matching main after #1016, and the branch is rebased onto main 8c3527b, where it compiles and its tests pass.

JingsongLi

Additional shared-foundation finding: the newly inherited expire_snapshots SQL timestamp parser is byte-identical to #965. An isolated real SQL test at #965 reproduces the Java DST-gap mismatch below; the existing current-main compile finding still applies.

JingsongLi · 2026-10-04T11:31:34Z

+                .and_then(|date| date.and_hms_opt(0, 0, 0))
+        })
+        .ok_or_else(|| DataFusionError::Plan(format!("Invalid older_than timestamp: '{value}'")))?;
+    Local


[P2] Match Java's forward resolution of DST gap timestamps

With TZ=America/New_York, CALL sys.expire_snapshots(table => 'test_db.t1', older_than => '2024-03-10 02:30:00', retain_min => 1) fails with 'does not exist in the local time zone'. Java ProcedureUtils uses DateTimeUtils.parseTimestampData with the default time zone; LocalDateTime.atZone resolves this gap forward to 03:30 EDT (1710055800000). This rejects a valid Java procedure input instead of expiring normally. Use the same gap-forward / fold-earlier resolution already implemented for timestamp options, and cover this input in an isolated time-zone test.

Fixed in the shared code of #965 (22d9e7e). older_than now uses main's Java-compatible local-timestamp resolution, so a DST-gap time moves forward and a fold takes the earlier instant. The test is pinned to TZ=America/New_York.

Port Java's ExpireSnapshotsImpl and SnapshotDeletion. Snapshots are chosen with snapshot.num-retained.min/max, snapshot.time-retained (a snapshot expires once its successor is older than the cut-off), consumer protection, and snapshot.expire.limit. Expiring [earliest, end) deletes data files removed by the deltas of (earliest, end] that the closest earlier tag does not read, changelog files, and manifest lists, manifests, index files, statistics and reassign plans that neither end nor a tag in the range references, then the snapshot files and finally moves the EARLIEST hint. Read failures only ever make a run delete less. Expose it as Table::new_expire_snapshots() and the DataFusion procedure sys.expire_snapshots with Java's arguments.

…ation Extend the file-set invariant to changelog files and index files kept in bucket directories, and add cases for deletion-vector files in both index layouts, changelog and hash index files of a primary-key table, external data files, unreadable deltas, tags and skipping sets (each must delete less, never more), the closest earlier tag among several, a snapshot missing from the range, the slowest of several consumers, and older_than given as a timestamp string.

…iring Expiration only protected the current branch's tags and the retained end snapshot, so expiring main deleted manifests and data files that a branch created from it still read, and could leave a long-lived changelog (changelog/changelog-<id>, written by Java's decoupled expiration before it removes the snapshot) without its manifest lists or changelog files. Before deleting anything, read the snapshots and tags of every branch and the long-lived changelogs of main and every branch. Their live data files join the protected set, their manifests (changelog manifests included) join the skipping set, and an expired snapshot's changelog files stay while any of them still names its changelog list. If these owners cannot be read, the run fails with nothing changed. Also resolve legacy deletion-vector locations before deleting index files: older Python writers left vectors under the table index directory even with index-file-in-data-file-dir, where reads find them but expiration did not.

…tory The skipping set of branches and long-lived changelogs, which reads their index and changelog manifests, was built only after data and changelog files had been deleted. An unreadable branch index manifest therefore failed the run after files of retained snapshots were gone. Build it right after reading the owners, before any deletion, so such a run fails with nothing changed. Resolve bucket directories from Table::data_file_location(), like writers, readers and commit cleanup do, instead of the table location: with data-file.path-directory set, expired data files, sidecars and bucket-local index files were left behind. Manifests, statistics and global index files stay under the table location.

apache#1016 removed the legacy deletion-vector location fallback (resolve_legacy_deletion_vector_entries) from reads, so expiration no longer compiled on main. Resolve index files only through committed_index_file_path, the layout reads use now, and drop the legacy-location test.

older_than rejected a local time inside a DST gap, which Java's DateTimeUtils.parseTimestampData moves forward (LocalDateTime.atZone). Generalize the Java-compatible scan.timestamp parser into spec::parse_local_timestamp_millis and use it for older_than, so a gap resolves forward and a fold to the earlier instant. Test both in a child process pinned to America/New_York.

Add PyTable.expire_snapshots(retain_max, retain_min, older_than_ms, max_deletes) on top of the core snapshot expiration. Unset arguments fall back to the table options, as in CALL sys.expire_snapshots, and the call releases the GIL while it runs.

Expire main through the Python API while a branch shares snapshot 1's files, and check both main and the branch still read their rows.

Create a table with data-file.path-directory, overwrite it and expire through the Python API; the expired file under data/bucket-0 is deleted and the table reads the new row.

zhuxiangyi · 2026-10-04T15:21:20Z

Both inherited findings are fixed in #965 (5e9cbad, 22d9e7e); details are in the inline replies. The branch is rebased onto it and main 8c3527b. make install && pytest tests/test_table.py passes (26 tests) on that merge, and so does clippy with -D warnings.

After the rebase, round 2's ad38d25 is 0201745.

zhuxiangyi mentioned this pull request Sep 26, 2026

[Feature] Table maintenance: snapshot expiration and orphan file cleanup #964

Open

2 of 6 tasks

zhuxiangyi force-pushed the feat/python-expire-snapshots branch from 114cc23 to 40eb555 Compare September 26, 2026 13:11

This was referenced Oct 1, 2026

feat(table): expire snapshots after commit #966

Open

feat(table): expire snapshots and add sys.expire_snapshots #965

Open

zhuxiangyi force-pushed the feat/python-expire-snapshots branch from 40eb555 to 13b76a2 Compare October 2, 2026 14:47

zhuxiangyi force-pushed the feat/python-expire-snapshots branch from 13b76a2 to 16a8c31 Compare October 2, 2026 15:20

JingsongLi requested changes Oct 3, 2026

View reviewed changes

JingsongLi reviewed Oct 3, 2026

View reviewed changes

zhuxiangyi force-pushed the feat/python-expire-snapshots branch from 16a8c31 to ad38d25 Compare October 3, 2026 13:15

JingsongLi reviewed Oct 4, 2026

View reviewed changes

zhuxiangyi added 9 commits October 4, 2026 22:43

test(python): expiring main keeps a branch readable

f867e7b

Expire main through the Python API while a branch shares snapshot 1's files, and check both main and the branch still read their rows.

test(python): expire under a configured data directory

0201745

Create a table with data-file.path-directory, overwrite it and expire through the Python API; the expired file under data/bucket-0 is deleted and the table reads the new row.

zhuxiangyi force-pushed the feat/python-expire-snapshots branch from ad38d25 to 0201745 Compare October 4, 2026 15:20

Conversation

zhuxiangyi commented Sep 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Brief change log

Tests

API and Format

Documentation

Uh oh!

JingsongLi commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhuxiangyi commented Oct 2, 2026

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

JingsongLi Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

zhuxiangyi Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

JingsongLi Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

zhuxiangyi Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

zhuxiangyi commented Oct 3, 2026

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

JingsongLi Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhuxiangyi Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

JingsongLi Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

zhuxiangyi Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

zhuxiangyi commented Oct 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhuxiangyi commented Sep 26, 2026 •

edited

Loading

JingsongLi commented Oct 1, 2026 •

edited

Loading

JingsongLi Oct 4, 2026 •

edited

Loading