[Arrow] Add API to check if Field has a valid ExtensionType

[sdf-jkl]

Which issue does this PR close?

• Closes [Arrow] Consider faster way to check if a Field has an extension type #8474.

Rationale for this change

Check issue

What changes are included in this PR?

• Added a has_valid_extension_type API to Field
• Added a unit test to save behavior

Are these changes tested?

• yes, unit test

Are there any user-facing changes?

[sdf-jkl]

@alamb @scovich I was looking at #8474 and the doc update in #8475.

The current recommendation in #8475 is good:

if field.extension_type_name() == Some(MyExtensionType::NAME) {
    if let Ok(extension_type) = field.try_extension_type::<MyExtensionType>() {
        // ...
    }
}

Checking the name first avoids the two name-related error paths in try_new_from_field_metadata (missing/mismatch).

For a full validity check, though, we currently still have to go through try_new_from_field_metadata, which means:

1. deserialize_metadata(...)
2. try_new(...)

Both may return Err, and both are part of validation today.

I think a clean follow-up would be a dedicated validate API on ExtensionType, with a default implementation that simply delegates to try_new_from_field_metadata (or equivalent). That gives us a clearer API now without changing behavior, and leaves room for specialized implementations later if needed. For extensions that override it, this could also avoid allocation/work from constructing an ExtensionType when we only need validation.

[sdf-jkl]

I could add ExtensionType::validate here

[sdf-jkl]

It was easy to do right away here.

Now instead of building the ExtensionType we only follow the simple validation steps from ExtensionType::try_new.

For these types (Bool8, Json, Uuid, Opaque, TimestampWithOffset, VariantType, RowNumber, RowGroupIndex) it's a simple DataType check.

FixedShapeTensor and VariableShapeTensor don't implement validate yet, but they don't really need it since we don't do a boolean check for them anywhere.

[scovich]

Generally LGTM, but one design question.

[scovich]

I fear we have an API clash here:

• supports_data_type receives &self, in order to allow configurable extension types based on their metadata
• validate receives metadata but can only use it by instantiating Self (which requires the very allocation we wanted to avoid).

Ideally, supports_data_type should be implemented in terms of validate instead... but I guess that would be a breaking change?

The next best would be to chase down every extension type that actually has metadata, and implement the two methods in the correct direction.

Does any impl in the arrow crate actually use this provided method? Or is it just a safety net for third party impl to avoid a breaking change?

[sdf-jkl]

It's just a safety net now. No extension type is using the default validate impl

[scovich]

Ack, thanks!

[alamb]

How about as a follow on we remove the default impl (and we merge it for the next major breaking API release)-- that will force any implementations to implement and avoid the potential slowdown

[sdf-jkl]

Sounds good!

[sdf-jkl]

filed here:

• [Arrow] Integrate ExtensionType::validate within ExtensionType::try_new #9708

[scovich]

LGTM, but I'd appreciate if @tustvold or @alamb could give a quick sanity check on the API design since they've been much more involved with extension types than I have.

[scovich]

Ack, thanks!

[alamb]

Thank you @scovich and @sdf-jkl