Fixed calculating column chunk statistics for empty array values

[norberttech]

Change Log

Added

Fixed

• calculating column chunk statistics for empty array values

Changed

Removed

Deprecated

Security

---

Description

I noticed a problem when saving lists that has last element empty:

(new Writer())
    ->write(
        __DIR__ . '/test.parquet',
        Schema::with(NestedColumn::list('primary_status_reasons', ListElement::string())),
        [
            [
                'primary_status_reasons' => ['NO_KEYWORDS']
            ],
            [
                'primary_status_reasons' => ['NO_KEYWORDS']
            ],
            [
                'primary_status_reasons' => []
            ]
        ]
    );

After quick debugging, it turned out that because of how we calculate total values in column chunk statistics, the page builder decided to split rows into chunks, where the first chunk had the first 2 rows and the last one, that empty row.

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| benchmark             | subject           | revs | its | mem_peak        | mode             | rstdev          |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 3.915mb +0.01%  | 508.606ms -0.10% | ±2.70% +686.88% |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 3.948mb +0.01%  | 1.071s +0.29%    | ±0.51% -64.70%  |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 28.508mb +0.00% | 431.133ms +0.17% | ±0.62% -12.65%  |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 3.675mb +0.01%  | 34.269ms +0.88%  | ±0.99% -8.87%   |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 3.622mb +0.01%  | 434.757ms -0.33% | ±0.86% +268.88% |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+

Transformers

+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| benchmark                   | subject                  | revs | its | mem_peak         | mode            | rstdev         |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 115.967mb +0.00% | 61.531ms +1.52% | ±0.64% -44.93% |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+----------------+

Loaders

+--------------------+----------------+------+-----+------------------+-----------------+-----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode            | rstdev          |
+--------------------+----------------+------+-----+------------------+-----------------+-----------------+
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 54.070mb +0.00%  | 85.367ms +0.14% | ±1.50% +41.43%  |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 106.503mb +0.00% | 53.335ms +1.34% | ±0.84% +262.34% |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 123.785mb +0.00% | 1.226s -0.56%   | ±0.36% +19.84%  |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 16.864mb +0.00%  | 44.466ms -0.75% | ±0.13% -90.09%  |
+--------------------+----------------+------+-----+------------------+-----------------+-----------------+

Building Blocks

+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak         | mode             | rstdev          |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 59.696mb +0.00%  | 435.517ms -5.75% | ±0.05% -98.42%  |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 14.235mb +0.00%  | 92.720ms +7.98%  | ±3.25% +6.32%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 116.517mb +0.00% | 499.794ms +0.43% | ±2.86% -7.26%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 59.996mb +0.00%  | 250.456ms -0.35% | ±0.67% -68.12%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 14.929mb +0.00%  | 56.607ms +6.09%  | ±2.25% +55.08%  |
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 86.787mb +0.00%  | 3.485ms +0.02%   | ±1.21% -12.94%  |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 102.385mb +0.00% | 189.041ms -0.58% | ±0.44% -68.49%  |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 85.105mb +0.00%  | 18.673ms -0.68%  | ±0.95% -21.73%  |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 88.027mb +0.00%  | 1.881ms -0.88%   | ±0.81% -12.79%  |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 88.027mb +0.00%  | 1.833ms -6.19%   | ±3.03% +76.98%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 85.139mb +0.00%  | 2.628ms -3.28%   | ±1.41% +31.56%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 85.668mb +0.00%  | 16.343ms -1.38%  | ±0.65% -20.94%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 85.668mb +0.00%  | 16.204ms -1.67%  | ±1.23% +98.29%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 83.573mb +0.00%  | 1.706μs -14.70%  | ±2.72% +0.00%   |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 83.573mb +0.00%  | 0.300μs -25.00%  | ±0.00% -100.00% |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 92.922mb +0.00%  | 12.149ms -0.37%  | ±0.38% -47.39%  |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 122.293mb +0.00% | 61.847ms +0.12%  | ±1.24% +122.07% |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 86.188mb +0.00%  | 1.318ms -12.75%  | ±1.10% -53.44%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 89.535mb +0.00%  | 59.313ms -5.94%  | ±0.97% -26.31%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 88.289mb +0.00%  | 4.007ms -3.85%   | ±1.29% +69.24%  |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 83.716mb +0.00%  | 38.975ms -3.52%  | ±0.73% -55.15%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 83.717mb +0.00%  | 39.050ms -4.99%  | ±0.57% -62.54%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 83.716mb +0.00%  | 39.114ms -1.99%  | ±0.99% -53.08%  |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 86.014mb +0.00%  | 7.370ms -0.46%   | ±0.46% +75.23%  |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 83.573mb +0.00%  | 28.754ms -2.20%  | ±0.79% -27.80%  |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 83.573mb +0.00%  | 13.688μs -0.48%  | ±0.69% -78.73%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 83.573mb +0.00%  | 16.200μs +2.31%  | ±1.01% -61.43%  |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 102.387mb +0.00% | 189.196ms -1.39% | ±0.50% -53.84%  |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+