[EPIC][Parquet] Finalize Variant Type support in Parquet

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Parquet adopted the Variant type from Spark VariantEncoding.md and we completed an initial implementation of Variant:

• [EPIC] [Parquet] Implement Variant type support in Parquet #6736

This support includes:

1. APIs for reading/writing Variant encoded data (parquet-variant crate)
2. APIs for converting to/from JSON (parquet-variant-json crate)
3. APIs for working with Arrow arrays of Variants, including kernels cast_to_variant, variant_get and shred_variant

Describe the solution you'd like
There are still several areas for additional work, with tickets gathered below

API Finalization

• [Variant] Rename variant_experimental flag to variant and remove warnings about being experimental #8297
• [Variant] Allow configurable JSON format for specialized types #8605
• [Variant] Remove ceremony of going from list of Variant to VariantArray #8606
• [Variant] Remove potential panics when probing VariantArray #8609
• [Variant] impl PartialEq for VariantArray #8610
• [Variant] Make VariantArray iterable #8612

Shredding Support

• [Variant] Support Shredded Objects in variant_get: untyped path access #8151
• [Variant] extend shredded null handling for arrays #8400
• [Variant] Support Shredded Objects in variant_get: access as Some(DataType::Struct) (nested shredding) #8153
• [Variant] Support Uuid/FixedSizeBinary(16) shredding #8665
• [Variant] [Shredding] Support typed_access for Decimal128 #8332
• [Variant] [Shredding] Support typed_access for Time64(Microsecond) #8334
• [Variant] [Shredding] Support typed_access for Struct #8336
• [Variant] [Shredding] Support typed_access for List #8337

Full variant_get Support

• [Variant] Support Shredded Objects in variant_get: Access as VARIANT (Unshredding) #8154
• [Variant] Support Shredded Lists/Array in variant_get #8082
• [Variant] Casting errors behavior support in variant_get #8086
• [Variant] Support typed access for other types in variant_get #8087
• [Variant] Support array access when parsing VariantPath #8946

** Full variant_to_arrow support

• [Variant] variant_to_arrow types support #8477

Infrastructure

• [Variant] Support BinaryArray/LargeBinaryArray in addition to BinaryViewArray for Variant #8387
• [Variant] Overly aggressive inference of UUID values #8420
• [Variant] Optimize the object header generation logic in ObjectBuilder::finish #7978
• [Variant] Revisit validation cost of infallible iterators #7711
• [Variant] VariantBuilder (wrongly?) accepts 0+ variant values #7870
• [Variant] Avoiding extra splice in ObjectBuilder::finish if possible #7960
• [Variant] Unify the cast option usage in ParquentVariant #8873

[sdf-jkl]

Grouping everything that seems to be left in one place:

API Finalization:

• [Variant] VariantArray field API naming #10093
• [Variant] Improve ergonomics when using VariantBuilder #9113
• [Variant] Improve ergonomic for creating VariantArray #8628

variant_get support:

• [Variant] Support Shredded Objects in variant_get: access as Some(DataType::Struct) (nested shredding) #8153
• [Variant] Support Shredded Objects in variant_get: Access as VARIANT (Unshredding) #8154
• [Variant] Support extracting struct fields as Variant using ExtensionType #9519
• [Variant] Align cast logic for variant_get to cast kernel #8982
  • Align cast logic betwee variant_get and cast kernel for utf8 type #10024
  • [Variant] Align cast logic between variant_get and cast kernel for temporal types (Timestamp/Date/Time) #10102
  • [Variant] When possible, cast perfectly shredded children #9862

variant_to_arrow:

• [Variant] variant_to_arrow types support #8477
  • [Variant] Add variant_to_arrow Map type support #10012
  • union?

Infrastructure:

• Fix the shred logic in parquet-variant #10145
• [Variant] Allow configurable JSON format for specialized types #8605
• [Variant] Support bulk-appends in cast_to_variant #8323
• [Variant] Overly aggressive inference of UUID values #8420
• [Variant] Consider consolidating struct builders between variant_get and shred_variant #9633
• [Variant] VariantBuilder (wrongly?) accepts 0+ variant values #7870
• Compilation failures due to conflicting versions of rand crate #9682
• [arrow-json] deserialize Variant fields #8987
• [Variant] Avoiding extra splice in ObjectBuilder::finish if possible #7960

The finale:

• [Variant] Rename variant_experimental flag to variant and remove warnings about being experimental #8297
• Blog post about Variant support #8566

cc @alamb @scovich @friendlymatthew @klion26 @liamzwbao @AdamGS

[sdf-jkl]

The checklist above is mostly complete and the remaining items were getting hard to see, so I gathered what is left into two focused epics: #10546 for the work gating the flag switch (#8297) and #10547 for non-blocking improvements. Leaving it to you whether this one stays open as the umbrella or gets closed in their favor.

[alamb]

Thanks @sdf-jkl -- I'll close this one then