Skip to content

Add Metadata::retain to filter entries in place #10683

Description

@alamb

Is your feature request related to a problem or challenge?

While updating DataFusion to arrow 60 I found the new Metadata struct introduced in #10075 has no way to remove entries based on a predicate. Code that previously called HashMap::retain on field/schema metadata now has to convert to a BTreeMap, filter, and convert back.

For example, DataFusion computes the intersection of metadata across UNION inputs, which now looks like:

let mut intersected: Option<BTreeMap<String, String>> = None;

for metadata in metadatas {
    match &mut intersected {
        None => {
            // deep copy into a BTreeMap so we can filter it below
            intersected =
                Some(metadata.iter().map(|(k, v)| (k.clone(), v.clone())).collect());
        }
        Some(current) => {
            current.retain(|k, v| metadata.get(k) == Some(&*v));
        }
    }
}

intersected.map(Metadata::from).unwrap_or_default()

Describe the solution you'd like

A Metadata::retain(|k, v| ...) method, copy-on-write like the existing insert/remove. The code above would then stay in terms of Metadata and avoid the deep copy and round-trip:

let mut intersected: Option<Metadata> = None;

for metadata in metadatas {
    match &mut intersected {
        None => intersected = Some(metadata.clone()), // cheap clone
        Some(current) => current.retain(|k, v| metadata.get(k) == Some(&*v)),
    }
}

intersected.unwrap_or_default()

Describe alternatives you've considered

Converting through BTreeMap as shown above; it works but adds a copy and obscures the intent.

Additional context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions