[Variant] Allow configurable JSON format for specialized types

[friendlymatthew]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently, the JSON read/write methods enforce a fixed mapping for specialized variant types. For instance, timestamps are always serialized as JSON strings.

Describe the solution you'd like

To make this more flexible, it should be possible to pass an optional format configuration that allows users to control how specialized types (like timestamps or UUIDs) are serialized/deserialized.

Additional context

This feature is motivated by datafusion-variant, a higher-level library that integrates with Datafusion through a series of udfs

[friendlymatthew]

cc @scovich

[scovich]

I couldn't find anything in the variant spec about how to convert to/from JSON.

I could have sworn Spark (which had the first implementation of variant) had some language about recommended serde rules for variant subtypes that don't map directly to json types, but I can't find anything like that now.

Absent any rules or precedent there, I think it comes down to whether we think we can cleanly express the gamut of desired behaviors. Similar questions have previously arisen with json parsing in general, and have generally not progressed due to complexity and concerns about semantics:

• Add string coercion when decoding json #7453
• Option for JSON parser to return NULL field values on type mismatch #7230

Ironically, part of the reason those issues didn't progress is because of claims (hopes?) that variant support would solve the problem.