Fix filter_record_batch panics with empty struct array#7539
Merged
alamb merged 2 commits intoapache:mainfrom May 22, 2025
Merged
Fix filter_record_batch panics with empty struct array#7539alamb merged 2 commits intoapache:mainfrom
filter_record_batch panics with empty struct array#7539alamb merged 2 commits intoapache:mainfrom
Conversation
thorfour
force-pushed
the
filter-empty-record
branch
from
9c9c598 to
7587b6d
Compare
exists from the predicate. This prevents panics in situations where a stucts array has empty columns.
thorfour
force-pushed
the
filter-empty-record
branch
from
7587b6d to
bbad99f
Compare
Contributor
|
cc @alamb |
alamb
approved these changes
May 21, 2025
| } | ||
|
|
||
| #[test] | ||
| fn test_filter_empty_struct() { |
Contributor
There was a problem hiding this comment.
I verified that this test fails without the code in this PR:
thread 'filter::tests::test_filter_empty_struct' panicked at arrow-array/src/array/struct_array.rs:224:51:
cannot use StructArray::new_unchecked if there are no fields, length is unknown
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
I think this is additional fallout from a new panic we introduced in this PR from @westonpace
Contributor
Author
There was a problem hiding this comment.
That's correct however without that panic the bug was much more subtle as it created an invalid record batch that would panic when slicing elsewhere.
alamb
changed the title
StructArraysfilter_record_batch panics with empty struct array. #7538
alamb
changed the title
filter_record_batch panics with empty struct array. #7538filter_record_batch panics with empty struct array
Contributor
|
Thanks again @thorfour |
thorfour
added a commit
to polarsignals/arrow-rs
that referenced
this pull request
May 27, 2025
* unit test: filter empty struct * When filtering struct Array's always build with a length since one exists from the predicate. This prevents panics in situations where a stucts array has empty columns.
This was referenced Jun 20, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
This prevents a panic that can happen with empty struct arrays.
What changes are included in this PR?
Small unit test that exacerbates the panic by attempting to filter a struct array with no columns.
Fix was to call
StructArray:new_unchecked_with_lengthsince the predicate of the filter contains the number rows the array requires.Are there any user-facing changes?
No