Optimize auto drop partition perfomance

[zcoo]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

With multi-level partition keys, it is not efficient to handle large partition set because current implement should scan all partition names.

Solution

We may sort partition names by partition time key and thus directly drop partitions with time value older than expected.
No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[LiebingYu]

Hi @wuchong. I would like to do it, please assign it to me.

Brief description.

1. partitionsByTable's value(TreeSet<String>) inAutoPartitionManager is now sorted by the entire string. In case of multiple partition keys, we need to sort by the time partition key. To achieve this goal, we need a comparator when constructing the TreeSet.

    private static class PartitionComparator implements Comparator<String> {

        private final int timePartitionKeyIndex;

        public PartitionComparator(
                List<String> partitionKeys, AutoPartitionStrategy autoPartitionStrategy) {
            String autoPartitionKey =
                    partitionKeys.size() == 1 ? partitionKeys.get(0) : autoPartitionStrategy.key();
            timePartitionKeyIndex = partitionKeys.indexOf(autoPartitionKey);
        }

        @Override
        public int compare(String p1, String p2) {
            String p1TimePartitionValue = p1.split("\\$")[timePartitionKeyIndex];
            String p2TimePartitionValue = p2.split("\\$")[timePartitionKeyIndex];
            int ret = p1TimePartitionValue.compareTo(p2TimePartitionValue);
            if (ret != 0) {
                return ret;
            } else {
                return p1.compareTo(p2);
            }
        }
    }

2. dropPartitions inAutoPartitionManager does not need to traverse the entire TreeSet and can directly jump out when currentTime.compareTo(lastRetainPartitionTime) >= 0 is satisfied.
3. Add tests with multiple partition keys in AutoPartitionManagerTest.

[wuchong]

@LB-Yu Thanks for the detailed implementation proposal. One optimization we can make is to avoid parsing and extracting the time partition value on every comparison — this can be expensive.

Instead, we can extract the time partition value once and use it as the key when storing entries in the TreeMap. This will improve performance by reducing redundant computation during comparisons.

I assigned this issue to you.