Crate datasketches 

Apache® DataSketches™ Core Rust Library Component

This crate provides compact, mergeable summaries for answering queries over large data streams. It implements a subset of the algorithms available in the other Apache DataSketches language components.

Enabling sketches

Sketch implementations are opt-in Cargo features; this crate enables none by default. Enable only the algorithms an application uses:

cargo add datasketches --features hll,theta

Each feature exposes a same-named module. For example, hll exposes datasketches::hll and tdigest exposes datasketches::tdigest.

Choosing a sketch

• Use bloom for probabilistic membership queries.
• Use countmin for point-frequency estimates and frequencies for discovering heavy hitters.
• Use hll for fast distinct counts, cpc for compact serialized distinct counts, or theta when set operations are required.
• Use req or tdigest for ranks and quantiles. REQ targets configurable high- or low-rank accuracy; T-Digest emphasizes distribution tails.
• Use tuple when retained Theta keys need application-defined summaries.

See each module’s documentation for accuracy, memory, serialization, and update examples.

Cross-language hashing

Compatible serialization does not by itself make ordinary Rust Hash input compatible with Java, C++, or Go. Rust strings and slices include type-specific framing, and short integers require different widening rules for different sketch families. When sketches must represent the same updates across languages, use the wrappers in hash::value:

• raw_bytes for byte and string contents;
• canonical_float for floating-point values;
• sign_extend for short integers used with HLL and CPC;
• natural_extend for short integers used with Bloom filters.

Other DataSketches implementations skip empty strings rather than hashing them. Check for empty input before updating when that cross-language behavior is required.

Modules

bloombloom

Bloom Filter implementation for probabilistic set membership testing.

codec

Codec utilities for datasketches crate.

common

Data structures and functions that may be used across all the sketch families.

countmincountmin

CountMin sketch implementation for frequency estimation.

cpccpc

Compressed Probabilistic Counting sketch.

error

Error types for DataSketches operations.

frequenciesfrequencies

Frequency sketches for finding heavy hitters in data streams.

hash

Hashing support for sketches.

hllhll

HyperLogLog sketch implementation for cardinality estimation.

reqreq

Relative Error Quantiles (REQ) sketch.

tdigesttdigest

T-Digest implementation for estimating quantiles and ranks.

thetatheta and (theta or tuple)

Theta sketch implementation for cardinality estimation.

thetacommontheta or tuple

Data structures and functions that may be used across all the Theta sketch family.

tupletuple and (theta or tuple)

Tuple sketch implementation.