[cleanup] Remove fastutil dependency - #25413
Merged
Merged
Conversation
Replace fastutil collections with minimal open-addressing hash map/set implementations in pulsar-common using the same approach (primitive keys, Fibonacci hashing, backward-shift deletion). Replace fastutil pair types with Java records. Use java.util.TreeMap directly for sorted maps. New classes in org.apache.pulsar.common.util.collections: - Long2ObjectOpenHashMap, Long2IntOpenHashMap, Int2ObjectOpenHashMap - LongOpenHashSet, IntOpenHashSet - IntIntPair, ObjectIntPair (records) - Long2ObjectMap, Long2IntMap (interfaces) - LongObjConsumer (functional interface) Migrated consumers: - DrainingHashesTracker, PersistentStickyKeyDispatcherMultipleConsumers - Consumer, InMemoryRedeliveryTracker, PendingAcksMap - InMemoryDelayedDeliveryTracker, NegativeAcksTracker Also removes the pulsar-client-dependencies-minimized module and all fastutil shading/relocation configuration from shaded client builds.
lhotari
approved these changes
Mar 27, 2026
This was referenced Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pulsar-commonusing the same approach (primitive keys, Fibonacci hashing, backward-shift deletion)IntIntPair,ObjectIntPair) with Java recordsjava.util.TreeMapdirectly for sorted maps instead of fastutil'sLong2ObjectAVLTreeMap/Long2ObjectRBTreeMappulsar-client-dependencies-minimizedmodule and all fastutil shading/relocation configurationNew classes in
org.apache.pulsar.common.util.collections:Long2ObjectOpenHashMap,Long2IntOpenHashMap,Int2ObjectOpenHashMapLongOpenHashSet,IntOpenHashSetIntIntPair,ObjectIntPair(records)Long2ObjectMap,Long2IntMap(interfaces)LongObjConsumer(functional interface)Migrated 7 files across
pulsar-brokerandpulsar-client:DrainingHashesTracker,PersistentStickyKeyDispatcherMultipleConsumersConsumer,InMemoryRedeliveryTracker,PendingAcksMapInMemoryDelayedDeliveryTracker,NegativeAcksTrackerMotivation
Fastutil is a ~21MB jar but Pulsar only uses a handful of its classes (maps, sets, and pairs) in 7 files. The custom implementations are minimal, only covering the operations actually used, and follow the same open-addressing design to preserve performance characteristics.
Why open-addressing hash maps instead of
java.util.HashMapOpen-addressing with primitive key arrays (
long[],int[]) avoids boxing keys toLong/Integerand avoids allocatingEntryobjects entirely. A map with N entries has zero object overhead beyond the flat arrays — this is a meaningful saving overHashMapwhich allocates anEntrynode per key-value pair plus boxes every primitive key.Why
TreeMapinstead of fastutil'sLong2ObjectAVLTreeMap/Long2ObjectRBTreeMapUnlike open-addressing hash maps, tree maps inherently allocate a node object per entry to maintain the tree structure. Since each entry already requires a node object, avoiding the
long → Longboxing of the key saves only 16 bytes per node — a negligible overhead on top of the node object itself.java.util.TreeMapprovides the same algorithmic guarantees (red-black tree, O(log n) operations, sorted iteration,headMap/tailMapviews) with no practical performance difference, so there is no reason to carry a custom implementation.Documentation
doc-not-neededMatching PR in forked repository
No
Test plan
Long2ObjectOpenHashMapTest,Long2IntOpenHashMapTest,Int2ObjectOpenHashMapTest,HashSetTest)ready-to-test