Skip to content

[cleanup] Remove fastutil dependency - #25413

Merged
merlimat merged 3 commits into
apache:masterfrom
merlimat:remove-fastutil
Mar 27, 2026
Merged

[cleanup] Remove fastutil dependency#25413
merlimat merged 3 commits into
apache:masterfrom
merlimat:remove-fastutil

Conversation

@merlimat

@merlimat merlimat commented Mar 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replace fastutil collections with minimal open-addressing hash map/set implementations in pulsar-common using the same approach (primitive keys, Fibonacci hashing, backward-shift deletion)
  • Replace fastutil pair types (IntIntPair, ObjectIntPair) with Java records
  • Use java.util.TreeMap directly for sorted maps instead of fastutil's Long2ObjectAVLTreeMap/Long2ObjectRBTreeMap
  • Remove the pulsar-client-dependencies-minimized module and all fastutil shading/relocation configuration

New classes in org.apache.pulsar.common.util.collections:

  • Long2ObjectOpenHashMap, Long2IntOpenHashMap, Int2ObjectOpenHashMap
  • LongOpenHashSet, IntOpenHashSet
  • IntIntPair, ObjectIntPair (records)
  • Long2ObjectMap, Long2IntMap (interfaces)
  • LongObjConsumer (functional interface)

Migrated 7 files across pulsar-broker and pulsar-client:

  • DrainingHashesTracker, PersistentStickyKeyDispatcherMultipleConsumers
  • Consumer, InMemoryRedeliveryTracker, PendingAcksMap
  • InMemoryDelayedDeliveryTracker, NegativeAcksTracker

Motivation

Fastutil is a ~21MB jar but Pulsar only uses a handful of its classes (maps, sets, and pairs) in 7 files. The custom implementations are minimal, only covering the operations actually used, and follow the same open-addressing design to preserve performance characteristics.

Why open-addressing hash maps instead of java.util.HashMap

Open-addressing with primitive key arrays (long[], int[]) avoids boxing keys to Long/Integer and avoids allocating Entry objects entirely. A map with N entries has zero object overhead beyond the flat arrays — this is a meaningful saving over HashMap which allocates an Entry node per key-value pair plus boxes every primitive key.

Why TreeMap instead of fastutil's Long2ObjectAVLTreeMap/Long2ObjectRBTreeMap

Unlike open-addressing hash maps, tree maps inherently allocate a node object per entry to maintain the tree structure. Since each entry already requires a node object, avoiding the long → Long boxing of the key saves only 16 bytes per node — a negligible overhead on top of the node object itself. java.util.TreeMap provides the same algorithmic guarantees (red-black tree, O(log n) operations, sorted iteration, headMap/tailMap views) with no practical performance difference, so there is no reason to carry a custom implementation.

Documentation

  • doc-not-needed

Matching PR in forked repository

No

Test plan

  • Unit tests added for all new collection classes (Long2ObjectOpenHashMapTest, Long2IntOpenHashMapTest, Int2ObjectOpenHashMapTest, HashSetTest)
  • Verify existing broker and client tests pass with the new implementations
  • ready-to-test

Replace fastutil collections with minimal open-addressing hash map/set
implementations in pulsar-common using the same approach (primitive keys,
Fibonacci hashing, backward-shift deletion). Replace fastutil pair types
with Java records. Use java.util.TreeMap directly for sorted maps.

New classes in org.apache.pulsar.common.util.collections:
- Long2ObjectOpenHashMap, Long2IntOpenHashMap, Int2ObjectOpenHashMap
- LongOpenHashSet, IntOpenHashSet
- IntIntPair, ObjectIntPair (records)
- Long2ObjectMap, Long2IntMap (interfaces)
- LongObjConsumer (functional interface)

Migrated consumers:
- DrainingHashesTracker, PersistentStickyKeyDispatcherMultipleConsumers
- Consumer, InMemoryRedeliveryTracker, PendingAcksMap
- InMemoryDelayedDeliveryTracker, NegativeAcksTracker

Also removes the pulsar-client-dependencies-minimized module and all
fastutil shading/relocation configuration from shaded client builds.
@merlimat merlimat changed the title Remove fastutil dependency (~21MB jar) [cleanup] Remove fastutil dependency (~21MB jar) Mar 27, 2026
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Mar 27, 2026
@merlimat merlimat changed the title [cleanup] Remove fastutil dependency (~21MB jar) [cleanup] Remove fastutil dependency Mar 27, 2026
@merlimat
merlimat merged commit 3d38403 into apache:master Mar 27, 2026
45 checks passed
@merlimat
merlimat deleted the remove-fastutil branch March 27, 2026 22:14
@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-not-needed Your PR changes do not impact docs ready-to-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants