Boolean encoding and RLEBitPackedHybrid

[norberttech]

Change Log

Added

• ConvertedType to column definitions in parquet for compatibility with other readers
• Repetitions/Definitions levels encodings in DataPage

Fixed

• Bug in RLEBitPackHybrid algorithm that was always bitpacking values

Changed

• Boolean columns can't be anymore dictionary packed for compatibility with spark

Removed

Deprecated

Security

---

Closes: #695

Description

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+-------------------+------+-----+------------------+-------------------+------------------+
| benchmark             | subject           | revs | its | mem_peak         | mode              | rstdev           |
+-----------------------+-------------------+------+-----+------------------+-------------------+------------------+
| AvroExtractorBench    | bench_extract_10k | 1    | 3   | 44.091mb +0.00%  | 351.553ms -17.69% | ±0.60% +1110.81% |
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 13.976mb +0.00%  | 263.647ms -21.05% | ±0.82% +45.31%   |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 18.637mb +0.00%  | 563.662ms -13.86% | ±0.58% +266.65%  |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 237.738mb +0.01% | 763.231ms -15.05% | ±0.58% +120.54%  |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 7.245mb +0.00%   | 14.561ms -22.16%  | ±0.68% +13.74%   |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 7.593mb +0.00%   | 447.718ms -20.60% | ±0.55% +291.84%  |
+-----------------------+-------------------+------+-----+------------------+-------------------+------------------+

Transformers

+-----------------------------+--------------------------+------+-----+-----------------+------------------+---------------+
| benchmark                   | subject                  | revs | its | mem_peak        | mode             | rstdev        |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+---------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 87.031mb +0.00% | 49.176ms -21.35% | ±0.72% +0.46% |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+---------------+

Loaders

+--------------------+----------------+------+-----+------------------+-------------------+-----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode              | rstdev          |
+--------------------+----------------+------+-----+------------------+-------------------+-----------------+
| AvroLoaderBench    | bench_load_10k | 1    | 3   | 93.195mb +0.00%  | 564.088ms -15.24% | ±1.35% +219.83% |
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 47.109mb +0.00%  | 69.360ms -0.87%   | ±0.77% +100.72% |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 88.543mb +0.00%  | 55.798ms -15.29%  | ±1.10% +90.70%  |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 281.637mb +0.00% | 1.220s -20.97%    | ±0.29% -54.13%  |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 16.530mb +0.00%  | 41.022ms +13.69%  | ±0.95% -44.31%  |
+--------------------+----------------+------+-----+------------------+-------------------+-----------------+

Building Blocks

+-------------------------+----------------------------+------+-----+-----------------+-------------------+------------------+
| benchmark               | subject                    | revs | its | mem_peak        | mode              | rstdev           |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+------------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 60.655mb +0.00% | 2.262ms -29.05%   | ±1.22% -65.50%   |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 80.448mb +0.00% | 150.935ms -15.06% | ±0.73% +45.02%   |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 58.974mb +0.00% | 14.933ms -15.62%  | ±0.39% +31.31%   |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 59.794mb +0.00% | 1.910ms -27.36%   | ±3.02% +89.86%   |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 59.794mb +0.00% | 1.797ms -33.18%   | ±2.33% +19.30%   |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 59.008mb +0.00% | 2.617ms -25.97%   | ±1.39% +1073.46% |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 59.537mb +0.00% | 14.303ms -18.07%  | ±1.55% -14.67%   |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 59.536mb +0.00% | 14.355ms -16.65%  | ±1.38% +368.74%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 57.607mb +0.00% | 1.800μs -18.17%   | ±0.00% -100.00%  |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 57.607mb +0.00% | 0.400μs -20.00%   | ±0.00% +0.00%    |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 65.841mb +0.00% | 10.319ms -25.62%  | ±0.28% -90.55%   |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 91.360mb +0.00% | 48.915ms -14.59%  | ±1.28% +56.25%   |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 60.057mb +0.00% | 2.140ms -24.25%   | ±1.98% +450.55%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 62.327mb +0.00% | 33.125ms -17.77%  | ±0.90% +36.01%   |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 62.158mb +0.00% | 4.776ms -29.29%   | ±2.76% +24.51%   |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 57.607mb +0.00% | 36.745ms -19.74%  | ±0.68% +164.78%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 57.607mb +0.00% | 36.817ms -19.29%  | ±0.50% +152.21%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 57.607mb +0.00% | 37.277ms -20.08%  | ±0.20% +16.76%   |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 59.882mb +0.00% | 7.350ms -23.83%   | ±0.41% -64.25%   |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 57.607mb +0.00% | 28.410ms -21.69%  | ±0.18% -42.38%   |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 57.607mb +0.00% | 13.099μs -36.43%  | ±1.87% +718.43%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 57.607mb +0.00% | 15.400μs -41.23%  | ±0.53% +195.05%  |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 80.448mb +0.00% | 156.877ms -11.26% | ±0.27% +6.47%    |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 91.722mb -0.01% | 117.528ms -16.65% | ±1.03% +140.00%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 47.591mb +0.01% | 61.955ms -13.78%  | ±2.41% +262.62%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 12.383mb +0.05% | 13.509ms -21.54%  | ±2.96% +593.92%  |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+------------------+