[Feature] Support reading Bitmap and HLL directly from VARBINARY #69912

Search before asking

• I had searched in the issues and found no similar issues.

Description

Doris should support converting raw VARBINARY values directly into BITMAP and HLL values.

The existing Hive Bitmap/HLL integration documented in the Hive UDF documentation relies on Doris Bitmap/HLL serialized data being consumed by the existing Base64-oriented functions, such as bitmap_from_base64 and hll_from_base64.

However, external binary type mapping has changed:

• PR #57821 introduced enable.mapping.varbinary for external binary columns;
• PR #68786 made external binary mapping unconditional and migrated legacy catalog settings;
• Hive/HMS BINARY, as well as supported Parquet/ORC binary columns, are now exposed as Doris VARBINARY while preserving their raw bytes.

The Doris Hive UDF implementations for to_bitmap/bitmap_union and to_hll/hll_union produce raw serialized byte arrays. With the current mapping behavior, those values are read as VARBINARY, while the existing functions expect Base64 text (STRING). Therefore, the documented Hive UDF workflow can no longer be used directly:

bitmap_from_base64(binary_column)
hll_from_base64(binary_column)

The type and encoding no longer match. Users should not have to convert the raw bytes to Base64 text first merely to convert them back into a Doris Bitmap or HLL.

Please add the following functions:

bitmap_from_binary(VARBINARY)  -> BITMAP
bitmap_to_binary(BITMAP)       -> VARBINARY
hll_from_binary(VARBINARY)     -> HLL
hll_to_binary(HLL)             -> VARBINARY

The *_from_binary functions should validate the complete input buffer and return row-level NULL for empty, truncated, or malformed values instead of reading beyond the input buffer or failing the entire query. The Bitmap path should also validate Roaring container contents because the input may come from an external user-controlled binary column.

Use case

This feature is needed for the following scenarios:

1. Read existing Hive UDF output

   Hive to_bitmap/bitmap_union and to_hll/hll_union output Doris-compatible serialized binary values. Doris should be able to read those values directly from Hive BINARY columns through an external catalog.

2. Read Parquet/ORC binary columns

   Parquet and ORC files can store Bitmap/HLL values as binary data. Since those columns are exposed as VARBINARY, users need a direct conversion function to query them with bitmap_count, bitmap_union, hll_cardinality, and related functions.

3. Write Doris values to external binary columns

   Users need to write Doris Bitmap/HLL values to Hive, Parquet, or ORC binary columns without an unnecessary Base64 text conversion.

4. Avoid a Base64 conversion workaround

   A workaround such as the following adds CPU, memory, intermediate strings, and data expansion:

   bitmap_from_base64(to_base64_binary(binary_column))

   Direct binary functions would preserve the original bytes and express the intended conversion clearly in SQL.

5. Safe handling of malformed external data

   External binary columns may contain empty, truncated, or corrupted values. Invalid rows should become NULL and must not cause out-of-bounds reads, heap corruption, or backend crashes during subsequent Bitmap/HLL operations.

Related issues

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct