[Variant] add variant_to_arrow union builder

[sdf-jkl]

Which issue does this PR close?

The last type to be supported by variant_to_arrow cast.

• Closes [Variant] variant_to_arrow types support #8477.

Rationale for this change

Check issue

What changes are included in this PR?

• Add variant_to_arrow union cast support + unit tests

Are these changes tested?

• Yes, unit tests

Are there any user-facing changes?

• Users can now cast Variant to Union type

[Jefffrey]

disclaimer: this is kinda my first time looking at variant code 😅

[Jefffrey]

do we need to consider nullability of the field?

[sdf-jkl]

I checked the existing Arrow union policy. UnionBuilder::append_null intentionally writes nulls into child arrays, while UnionBuilder::build still creates those child Fields with nullable = false. UnionArray::try_new does not validate field nullability, and union nullness is derived from the selected child array. Therefore the fallback does not need to select a nullable-declared field.

arrow-rs/arrow-array/src/builder/union_builder.rs

Lines 285 to 317 in e28fd0d

	let mut children = Vec::with_capacity(self.fields.len());
	let union_fields = self
	.fields
	.into_iter()
	.map(
	|(
	name,
	FieldData {
	type_id,
	data_type,
	mut values_buffer,
	slots,
	mut null_buffer_builder,
	},
	)| {
	let array_ref = make_array(unsafe {
	ArrayDataBuilder::new(data_type.clone())
	.add_buffer(values_buffer.finish())
	.len(slots)
	.nulls(null_buffer_builder.finish())
	.build_unchecked()
	});
	children.push(array_ref);
	(type_id, Arc::new(Field::new(name, data_type, false)))
	},
	)
	.collect();
	UnionArray::try_new(
	union_fields,
	self.type_id_builder.into(),
	self.value_offset_builder.map(Into::into),
	children,
	)

[Jefffrey]

the builder thing is a known issue:

• UnionBuilder produces incorrect Union DataType #1637

i guess its one of those things which is a bit messy in the codebase as is 🤔

[sdf-jkl]

Thanks @Jefffrey 🙏

This is not the most Varianty code, mostly about understanding the Union type

[sdf-jkl]

@klion26 please you take a look 🙏

[Jefffrey]

thanks @sdf-jkl

[klion26]

Thanks for pushing this forward. Sorry for the late reply, a little busy in the inter works the last days. I'll take a look at this and reply if I have any comments

[klion26]

@sdf-jkl I've passed the code, and left some comments inline; thanks.

[klion26]

Here we use type-based info to determine which target the current variant should belong to. This seems incorrect

1. Variant::Int64(1) can't convert to DataType::Int8, but this should valid when calling variant_get
2. The logic in the current code (json -> variant, and variant spec), treats the exact number the same in Variant::Int64/Int32/Int16 etc(value has no strict type in variant) (like the Equivalence Class in parquet variant spec)

[klion26]

Maybe we can sort the field in the Union, and try the target column in the sorted order like we treat like the json-> variant(from_json.rs in parquet-variant-compute)