Repository navigation
Improve Arrow Parquet writer to include UUID logical type - #100150
Merged
alexey-milovidov merged 1 commit intoMar 20, 2026
Merged
Conversation
ivanmantova
force-pushed
the
fix-arrow-parquet-builder
branch
from
March 19, 2026 23:49
47ac08b to
ff8664c
Compare
ivanmantova
changed the title
ivanmantova
changed the title
alexey-milovidov
added
the
can be tested
Contributor
|
Workflow [PR], commit [ff8664c] Summary: ✅ AI ReviewSummaryThis PR updates Arrow-based UUID export by introducing Arrow UUID extension typing and a builder-safe conversion path ( ClickHouse Rules
Final Verdict
|
clickhouse-gh
Bot
added
the
pr-improvement
Contributor
LLVM Coverage Report
PR changed lines: PR changed-lines coverage: 92.00% (69/75, 0 noise lines excluded) |
Contributor
Author
|
Looks like the public CI is looking great, but the CH Inc sync check failed. Could a maintainer take a quick look? Let me know if it's related to the Arrow parquet writer UUID changes or not, and I'd be happy to push a fix if needed! Thanks! |
alexey-milovidov
approved these changes
Mar 20, 2026
alexey-milovidov
left a comment
alexey-milovidov
left a comment
Member
There was a problem hiding this comment.
Thanks for the quick fix!
robot-ch-test-poll4
added
the
pr-synced-to-cloud
fm4v
added a commit
that referenced
this pull request
Mar 21, 2026
…tion When a PR changes a C++ source file, tests that cover sibling files (same directory, same functional domain) are now included as additional candidates, ranked below direct hits (SIBLING_DIR_WIDTH=10000). Problem solved: for PR #100150 (Arrow writer UUID fix), the 16 directly selected tests all covered CHColumnToArrowColumn.cpp (the writer), but 32 additional tests cover ArrowColumnToCHColumn.cpp (the reader) which lives in the same directory and is functionally related. These tests were missed because they don't call the writer directly. Algorithm (new _query_sibling_dir_tests method): 1. Extract domain keywords from changed C++ source filenames using CamelCase word splitting, filtering out architectural/common words. E.g. CHColumnToArrowColumn.cpp → ["Arrow"] MergeTreeIndexConditionText.cpp → ["Index", "Text"] 2. With one keyword: LIKE '%Arrow%' (OR logic). With multiple: LIKE '%Index%' AND LIKE '%Text%' (AND logic avoids false matches on broad words like "Index" alone). 3. Inner query: find sibling files (same dir, keyword-filtered) that the PRIMARY tests already cover — these are files with a natural code relationship to the changed file. 4. Outer query: find tests covering those same sibling files that were NOT in the primary set — these tests exercise the same domain from a different entry point. Results: PR #100150 (Arrow writer): 16 → 48 tests (+32 Arrow reader tests) PR #100224 (text index): 71 → 112 tests (+41 text/bloom index tests) No false positives added for unrelated PRs. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
alexbakharew
added
pr-bugfix
8 of 27 tasks
zvonand
added a commit
to Altinity/ClickHouse
that referenced
this pull request
May 21, 2026
Antalya 26.3 Backport of ClickHouse#99521, ClickHouse#100150 - Add Arrow and Parquet format support for UUID data type
subkanthi
pushed a commit
to Altinity/ClickHouse
that referenced
this pull request
May 26, 2026
…et-builder Improve Arrow Parquet writer to include UUID logical type
subkanthi
pushed a commit
to Altinity/ClickHouse
that referenced
this pull request
May 26, 2026
Antalya 26.3 Backport of ClickHouse#99521, ClickHouse#100150 - Add Arrow and Parquet format support for UUID data type
il9ue
pushed a commit
to il9ue/ClickHouse
that referenced
this pull request
Jun 1, 2026
…et-builder Improve Arrow Parquet writer to include UUID logical type
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #100119
Changelog category:
Changelog entry:
Exporting UUIDs to Parquet via the Arrow encoder now includes the correct UUID type annotation, eliminating the need to manually cast
FixedString(16)data when reading the files back into ClickHouse or other systems.Motivation
When exporting to Parquet using the Arrow-based writer, UUID columns were previously written as raw
FIXED_LEN_BYTE_ARRAY(16)without thelogicalType.UUIDannotation. This caused downstream systems (and ClickHouse's own Parquet reader) to infer the column asFixedString(16)rather than a proper UUID.Ensures full data interoperability for UUIDs when exporting to Parquet via Arrow, so downstream tools read the type correctly out of the box without requiring manual schema hints.
Parameters
output_format_parquet_use_custom_encoder=0: Applies specifically when this setting is explicitly set to 0 (which routes Parquet generation through the Arrow subsystem). The native Parquet encoder (1, the default) is unaffected and already handles UUIDs correctly.Version info
26.4.1.66