Skip to content

[Variant] Add variant_to_arrow Struct type support#9572

Merged
alamb merged 5 commits into
apache:mainfrom
sdf-jkl:struct-variant_to_arrow
Mar 19, 2026
Merged

[Variant] Add variant_to_arrow Struct type support#9572
alamb merged 5 commits into
apache:mainfrom
sdf-jkl:struct-variant_to_arrow

Conversation

@sdf-jkl
Copy link
Copy Markdown
Contributor

@sdf-jkl sdf-jkl commented Mar 18, 2026

Which issue does this PR close?

Rationale for this change

  • In a follow up PR, can fix the variant_get TODO:
    let struct_array = field.as_struct_opt().ok_or_else(|| {
    // TODO: Should we blow up? Or just end the traversal and let the normal
    // variant pathing code sort out the mess that it must anyway be
    // prepared to handle?
  • When we know that Struct VariantArray is not shredded can reuse shred_basic_variant

What changes are included in this PR?

  • Added StructVariantToArrowRowBuilder builder.
  • Moved make_variant_to_arrow_row_builder logic to make_typed_variant_to_arrow_row_builder to reuse by Struct array's inner fields.
  • Changed a variant_get test to show that it now handles unshredded Struct VariantArray

Are these changes tested?

  • Yes, added test_struct_row_builder_handles_unshredded_nested_structs
  • Everything else still works.

Are there any user-facing changes?

No

@github-actions github-actions Bot added the parquet-variant parquet-variant* crates label Mar 18, 2026
@sdf-jkl sdf-jkl changed the title add variant_to_arrow Struct type support [Variant] Add variant_to_arrow Struct type support Mar 18, 2026
@sdf-jkl
Copy link
Copy Markdown
Contributor Author

sdf-jkl commented Mar 18, 2026

@klion26 @codephage2020 @scovich bothering you again... 😄

Copy link
Copy Markdown
Contributor

@scovich scovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with nits, thanks!

Comment thread parquet-variant-compute/src/variant_get.rs Outdated
Comment thread parquet-variant-compute/src/variant_to_arrow.rs Outdated
Comment thread parquet-variant-compute/src/variant_to_arrow.rs Outdated
sdf-jkl and others added 4 commits March 18, 2026 15:26
Co-authored-by: Ryan Johnson <scovich@users.noreply.github.com>
Co-authored-by: Ryan Johnson <scovich@users.noreply.github.com>
@sdf-jkl
Copy link
Copy Markdown
Contributor Author

sdf-jkl commented Mar 18, 2026

@scovich I've addressed the nits, thanks for the review!

Copy link
Copy Markdown
Member

@klion26 klion26 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGMT, thanks for the contribution.

@alamb alamb merged commit 55a7768 into apache:main Mar 19, 2026
17 checks passed
@alamb
Copy link
Copy Markdown
Contributor

alamb commented Mar 19, 2026

Love it -- thanks @sdf-jkl @klion26 and @scovich

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet-variant parquet-variant* crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Variant] Add variant_to_arrow Struct type support

4 participants