Repository navigation
feat: Add Arrow JSON integration testing executable. Fix ArrowReader bug. - #191
Open
willtemperley wants to merge 3 commits into
Open
willtemperley wants to merge 3 commits into
willtemperley wants to merge 3 commits into
Conversation
…hich was ignoring zero-length null buffers.
Open
Contributor
|
I tested the integration test executable on Debian 13 against generated_primitive from apache/arrow-testing (cpp-21.0.0 gold files).
$ swift run arrow-json-integration-test \ base 15:40:04
--arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
--json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
--mode=ARROW_TO_JSON
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.28s)
$ swift run arrow-json-integration-test \ base 15:40:09
--arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
--json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
--mode=JSON_TO_ARROW
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.28s)
Error: DecodingError.typeMismatch: expected value of type DataValue. Path: batches[0].columns[0].DATA[0]. Debug description: Cannot decode DataValue
$ swift run arrow-json-integration-test \ base 15:40:16
--arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
--json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
--mode=VALIDATE
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.29s)
Error: DecodingError.typeMismatch: expected value of type DataValue. Path: batches[0].columns[0].DATA[0]. Debug description: Cannot decode DataValueBut I believe this @kou LGTM |
kou
reviewed
Sep 14, 2026
| } | ||
| let schema = schemaBuilder.finish() | ||
|
|
||
| switch ArrowWriter().writeFile(ArrowWriter.Info(.recordbatch, schema: schema, batches: batches)) { |
Member
There was a problem hiding this comment.
Should we use toFile() instead of writeFile() to write the ARROW1 marker?
Comment on lines
+112
to
+119
| let expected = try JSONDecoder().decode( | ||
| ArrowJSON.self, from: try Data(contentsOf: jsonURL)) | ||
|
|
||
| let actual = try decodeArrowJSON(from: arrowURL) | ||
|
|
||
| guard expected.normalized() == actual.normalized() else { | ||
| throw ValidationError("Arrow file does not match JSON file") | ||
| } |
Member
There was a problem hiding this comment.
How about comparing in Apache Arrow data instead of JSON data?
Archery doesn't use the Apache Arrow -> JSON conversion. If we compare data in Apache Arrow, we don't need to implement Apache Arrow -> JSON conversion. The Apache Arrow data comparison will be also useful in other use cases.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's Changed
Added an Arrow-JSON integration testing executable as discussed in #96. This includes an ArrowJSON struct for decoding and encoding the Arrow JSON test files.
Added a rule to .swiftlint.yml to ignore case statements in cyclomatic complexity checks.
Made some Arrow Type declarations public to support encoding and decoding.
Fix: ArrowReader now creates a null buffer of the length specified by the flatbuffers definition, instead of calculating it from the array length. This fixes a bug where a null buffer of full length would be created and allocated as zeros, making a non-null array become all-null.