Skip to content

feat: Add Arrow JSON integration testing executable. Fix ArrowReader bug. - #191

Open
willtemperley wants to merge 3 commits into
apache:mainfrom
willtemperley:main
Open

willtemperley wants to merge 3 commits into
apache:mainfrom
willtemperley:main

Conversation

@willtemperley

Copy link
Copy Markdown
Contributor

What's Changed

Added an Arrow-JSON integration testing executable as discussed in #96. This includes an ArrowJSON struct for decoding and encoding the Arrow JSON test files.

Added a rule to .swiftlint.yml to ignore case statements in cyclomatic complexity checks.

Made some Arrow Type declarations public to support encoding and decoding.

Fix: ArrowReader now creates a null buffer of the length specified by the flatbuffers definition, instead of calculating it from the array length. This fixes a bug where a null buffer of full length would be created and allocated as zeros, making a non-null array become all-null.

@willtemperley willtemperley mentioned this pull request Sep 1, 2026
@maltzsama

Copy link
Copy Markdown
Contributor

I tested the integration test executable on Debian 13 against generated_primitive from apache/arrow-testing (cpp-21.0.0 gold files).
What works:

  • ARROW_TO_JSON
  • VALIDATE (on the .arrow_file format)
  • Build compiles cleanly (only deprecation warnings from flatbuffers and ArrowCExporter)
    What doesn't work yet (not a blocker for this PR):
  • JSON_TO_ARROW and VALIDATE on .arrow_file fail with:
$ swift run arrow-json-integration-test \                                                                                                                            base 15:40:04
  --arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
  --json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
  --mode=ARROW_TO_JSON
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.28s)
$ swift run arrow-json-integration-test \                                                                                                                            base 15:40:09
  --arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
  --json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
  --mode=JSON_TO_ARROW
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.28s)
Error: DecodingError.typeMismatch: expected value of type DataValue. Path: batches[0].columns[0].DATA[0]. Debug description: Cannot decode DataValue
$ swift run arrow-json-integration-test \                                                                                                                            base 15:40:16
  --arrow="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.arrow_file" \
  --json="/tmp/arrow-testing/data/arrow-ipc-stream/integration/cpp-21.0.0/generated_primitive.json" \
  --mode=VALIDATE     
Building for debugging...
[1/1] Write swift-version-7FB2667E20D8AC03.txt
Build of product 'arrow-json-integration-test' complete! (0.29s)
Error: DecodingError.typeMismatch: expected value of type DataValue. Path: batches[0].columns[0].DATA[0]. Debug description: Cannot decode DataValue

But I believe this DataValue in ArrowJSON.swift decode can be addressed in another issue.

@kou LGTM

}
let schema = schemaBuilder.finish()

switch ArrowWriter().writeFile(ArrowWriter.Info(.recordbatch, schema: schema, batches: batches)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use toFile() instead of writeFile() to write the ARROW1 marker?

Comment on lines +112 to +119
let expected = try JSONDecoder().decode(
ArrowJSON.self, from: try Data(contentsOf: jsonURL))

let actual = try decodeArrowJSON(from: arrowURL)

guard expected.normalized() == actual.normalized() else {
throw ValidationError("Arrow file does not match JSON file")
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about comparing in Apache Arrow data instead of JSON data?
Archery doesn't use the Apache Arrow -> JSON conversion. If we compare data in Apache Arrow, we don't need to implement Apache Arrow -> JSON conversion. The Apache Arrow data comparison will be also useful in other use cases.

@kou kou changed the title Add Arrow JSON integration testing executable. Fix ArrowReader bug. feat: Add Arrow JSON integration testing executable. Fix ArrowReader bug. Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants