Add `PageIndexBuilder` and `PageIndexProvider` for Parquet page indexes - #10842

etseidl · 2026-08-25T19:27:46Z

Which issue does this PR close?

Part of Support load page index metadata with specified column id instead of all columns #7582
Closes parquet: PageIndex cannot be constructed outside the crate #10824.

Rationale for this change

This grew out of a discussion in #10784 and relates to apache/datafusion#24288 (comment). This PR provides a new builder for creating page index structures, and also adds a new PageIndexProvider trait to allow more performant implementations.

What changes are included in this PR?

Adds PageIndexBuilder, PageIndexProvider, implements PageIndexProvider for PageIndex, and adds RowGroupPageIndex as a helper to support fetching page indexes for a specific row group (replaces the X_index_for_rowgroup() functions on PageIndex).

Are these changes tested?

Yes, should be covered by existing tests

Are there any user-facing changes?

Yes, this changes the public API for accessing page index information

Created with the aid of Claude Code, but I own the changes.

etseidl · 2026-08-29T19:42:32Z

Thanks @alamb. The biggest change from your last look is the example is more compelling now.

alamb

Looks good to me -- the example is especially nice now I think

Thank you @etseidl

I think we should avoid the clone in heap_size before merging, but everything else can be done as a follow on (I can help file tickets if you like):

Writer drops custom page indexes
memory accounting on the trait
doc cleanups

alamb · 2026-09-02T11:42:03Z

+//! - Uses nested HashMaps for efficient storage and lookup
+//! - Implements all required PageIndexProvider trait methods
+//!
+//! This approach can significantly reduce memory usage when working with wide


Can we also claim it reduces metadata load time?

"This approach can significantly reduce memory usage and metadata load time when working with wide
tables and you only access a few columns".

Though maybe we should also explain when page level statistics are helpful (evaluating predicates (stats) or fetching specific ranges of rows (after predicates or index application). That might be too nuanced however 🤔

done in 0caa8b4

alamb · 2026-09-02T11:42:53Z

+use std::sync::Arc;
+use tempfile::TempDir;
+
+//////////////////////////////////////////////


nit: my personal preference is to put helpers at the end so the example starts with the "punchline" and then people can refer to the details if they need. However I am not sure how important that is going forward with coding agents, etc

done in 0caa8b4

alamb · 2026-09-02T11:50:31Z

We can finally have nice things -- a module that has a page index separated -- so nice!

alamb · 2026-09-02T11:59:55Z

+        if let Some(page_index_arc) = self.metadata.page_index.as_ref()
+            && let Some(page_index) = page_index_arc
+                .as_any()
+                .downcast_ref::<crate::file::metadata::PageIndex>()


Downcast to PageIndex to access raw index structures for serialization

Claude points out that this means that the writer not write page indexes if the provider is a custom provider. I think we need to either explicitly call that out in docs, or (preferably) actually serialize the page indexes when sourced from a custom provider

We could document the limitation in this PR and then fix it in a follow on PR (I bet if we wrote up a ticket someone else would do it)

done in 690907c

need to file issue

Filed ParquetMetaDataWriter silently omits page indexes from custom PageIndexProviders #11030

alamb · 2026-09-02T12:03:39Z

+        // out of scope.
+        let page_index_size = if let Some(page_index) = self.page_index.as_ref() {
+            if let Some(page_index) = page_index.as_any().downcast_ref::<PageIndex>() {
+                let page_index = Some(Arc::new(page_index.clone()));


this clone deep copies the page index -- I think it should be something mor elike std::mem::size_of::<PageIndex>() + page_index.heap_size()

Yeah, I knew this was janky. I'm working through the correct thing to do here. I think this form was undercounting when the page index was Some anyway.

I want to push heap_size into the provider API now because I think adding a function to a public trait constitutes a breaking change. I'll see if I can get this right today.

fixed in 3e382e5

I did some manual accounting. For an unpopulated index, the size should go from the size of an Option<PageIndex> 48 bytes to Option<Arc<...>> 16 bytes so a net -32 bytes.

For a populated index, we need to add in the heap size of the index (450 bytes), the size of the now heap allocated PageIndex (48 bytes), plus 16 bytes of heap allocated Arc overhead. This adds 64 bytes to the heap size, but we saved 32 above, so we should net increase 32 bytes for the test case.

alamb · 2026-09-02T12:03:40Z

        #[cfg(not(feature = "encryption"))]
        let encryption_size = 0usize;

+        // We can only determine the heap size for PageIndex. Custom providers are


I think it is fine to not include heap size in the memory usage calculation in this PR, but we should file a follow on PR to add an API for a custom index provider to report its memory usage (as one of the main points of this PR is to have more efficient caching, for which we need to know how large the memory usage is)

added TODO in d562697

need to file an issue for this

filed Support reporting heap size of custom PageIndexProvider #11028

alamb · 2026-09-02T12:06:04Z

+/// [`OffsetIndex`]: crate::file::page_index::offset_index::OffsetIndexMetaData
+/// [`ColumnChunkMetaData`]: crate::file::metadata::ColumnChunkMetaData
+pub trait PageIndexProvider: Send + Sync + std::fmt::Debug {
+    /// Returns `true` if offset index structures are present


It is probably also worth mentioning here that if has_offset_indexes should return false only of offset_index will always return false -- aka if this method reports false, the reader/writer won't even try to load any offset indexes

As written it is slightly unclear if it should report false of no indexes are currently loaded but some might be loaded in the future

attempted fix in 39c5828

…o page_index_provider

etseidl · 2026-09-02T20:40:40Z

Thanks @alamb! I think I've addressed all of your comments, PTAL when you can. I still need to add issues for the serialization gap and memory accounting. I think the former is a pretty easy fix, but the latter will take some thought (we've gone down this rabbit hole before (e.g. #9138)

Edit: I have the serialization ready to go once this merges. Non-breaking so it can wait for 60.1.0.

bharath-techie · 2026-09-03T18:44:47Z

Thanks for the changes @etseidl

alamb

Thanks @etseidl -- I went through this one again

I filed some follow on tickets

And then I pushed some commits to:

Update the TODO comments with links to issues
Fix some other random small cleanups found while reviewing

I think we are ready now. Maybe you can take one last quick peek and then merge it in!

alamb · 2026-09-08T20:43:35Z

        #[cfg(not(feature = "encryption"))]
        let encryption_size = 0usize;

+        // We can only determine the heap size for PageIndex. Custom providers are


filed Support reporting heap size of custom PageIndexProvider #11028

etseidl · 2026-09-08T20:51:44Z

Thanks @alamb, your changes look good. I'll merge once CI finishes.

adriangb · 2026-09-08T23:46:57Z

Amazing to see this land! @etseidl do you plan to use this to work on alternative storage / representation for page indexes (particularly for wide tables with sparse reads)?

alamb · 2026-09-09T00:04:17Z

We did it!

etseidl · 2026-09-09T00:10:04Z

Amazing to see this land! @etseidl do you plan to use this to work on alternative storage / representation for page indexes (particularly for wide tables with sparse reads)?

Thanks @adriangb. TBH I haven't really thought that far ahead. I was hoping to get more requirements from the Datafusion side before doing too much further engineering. I'm certainly interested in having other providers in the parquet crate.

etseidl and others added 30 commits August 11, 2026 14:21

convert page indexes to Vec<Vec<Option<T>>>

1f5d629

remove the NONE variant from ColumnIndexMetaData

250644d

doc fix

511ffba

Merge branch 'main' into opt_page_index

03cbefc

Merge remote-tracking branch 'origin/main' into opt_page_index

2a93a11

Merge remote-tracking branch 'origin/main' into opt_page_index

6a45480

clippy

158ab8f

checkpoint

965c6a0

checkpoint

80a16ad

basic tests pass

b410401

fix test

1794e41

fix benches and bin

b971e38

fix more tests

78d869a

fix doc test

a7e424a

fix test for missing col idx

b9bf6e2

clippy

39469db

doc fixes

c76b016

remove fixmes

c47386d

clippy

7f47fac

claude some documentation

c6edf24

fix test todo

4315c9f

add PageIndex::is_complete per review suggestion

ae3d0d2

Merge branch 'main' into new_page_index

50041c8

fix merge

04b8f67

reword docs to be less implementation specific

a4b7303

Merge branch 'main' into new_page_index

beda053

Merge remote-tracking branch 'origin/main' into new_page_index

f236ad9

add page index builder

5f1b746

experiment with selective index population

fc465f6

fix docs

10a5eb2

Merge remote-tracking branch 'origin/main' into page_index_provider

6806d69

alamb approved these changes Sep 2, 2026

View reviewed changes

etseidl added 8 commits September 2, 2026 07:50

refactor and add comments per review suggestion

0caa8b4

fix memory_size calculation

3e382e5

add TODO

d562697

add warning and todo re serialization limited to PageIndex

690907c

add some clarification to docs for PageIndexProvider::has_XXX_indexes

39c5828

add some sugar to obtain RowGroupPageIndexes

36ec3a1

Merge branch 'page_index_provider' of github.com:etseidl/arrow-rs int…

410975a

…o page_index_provider

clippy

28009fa

etseidl commented Sep 2, 2026

View reviewed changes

Comment thread parquet/src/file/metadata/mod.rs

etseidl added 2 commits September 4, 2026 14:44

Merge remote-tracking branch 'origin/main' into page_index_provider

ea6eb07

spelling

79dbf5a

alamb mentioned this pull request Sep 8, 2026

Support reporting heap size of custom PageIndexProvider #11028

Open

Update link to filed issue

06a8bdd

This was referenced Sep 8, 2026

"end to end" test for reading parquet using custom PageIndexProvider #11029

Open

ParquetMetaDataWriter silently omits page indexes from custom PageIndexProviders #11030

Open

alamb added 3 commits September 8, 2026 16:41

add comments / update todo

3a906c4

Merge remote-tracking branch 'apache/main' into page_index_provider

4c7adb6

Clean up docs example, test names, urls

bc8fa39

alamb approved these changes Sep 8, 2026

View reviewed changes

etseidl merged commit 7438071 into apache:main Sep 8, 2026
32 checks passed

etseidl deleted the page_index_provider branch September 8, 2026 22:44

Uh oh!

Conversation

etseidl commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

etseidl commented Aug 29, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

bharath-techie commented Sep 3, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl commented Sep 8, 2026

Uh oh!

Uh oh!

adriangb commented Sep 8, 2026

Uh oh!

alamb commented Sep 9, 2026

Uh oh!

etseidl commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

etseidl commented Aug 25, 2026 •

edited

Loading

etseidl commented Sep 2, 2026 •

edited

Loading