Parquet performance

[norberttech]

Change Log

Added

Fixed

Changed

• Improved parquet writer performance

Removed

Deprecated

Security

---

Description

Before: https://blackfire.io/profiles/03764c79-8dbc-4924-956d-36324f3b5d00/graph
After V1: https://blackfire.io/profiles/c4f1c849-ecdf-49ab-ae4a-faeb0a744482/graph
After V2: https://blackfire.io/profiles/10ce027a-003e-4a1a-a03c-cf6e42c3ccd5/graph

Profiling was executed at 100k rows.

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+-------------------+------+-----+------------------+-------------------+----------------+
| benchmark             | subject           | revs | its | mem_peak         | mode              | rstdev         |
+-----------------------+-------------------+------+-----+------------------+-------------------+----------------+
| AvroExtractorBench    | bench_extract_10k | 1    | 3   | 44.091mb +0.00%  | 431.983ms -17.19% | ±0.10% -92.93% |
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 13.976mb +0.00%  | 336.512ms -18.63% | ±0.39% +74.71% |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 18.638mb +0.00%  | 666.253ms -20.28% | ±0.73% -44.87% |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 237.693mb +2.22% | 935.508ms -18.98% | ±0.31% +50.43% |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 7.245mb +0.01%   | 18.830ms -12.97%  | ±0.18% -81.71% |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 7.593mb +0.01%   | 568.026ms -21.71% | ±0.17% -74.35% |
+-----------------------+-------------------+------+-----+------------------+-------------------+----------------+

Transformers

+-----------------------------+--------------------------+------+-----+-----------------+------------------+-----------------+
| benchmark                   | subject                  | revs | its | mem_peak        | mode             | rstdev          |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+-----------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 87.032mb +0.00% | 63.270ms -18.67% | ±0.33% +340.42% |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+-----------------+

Loaders

+--------------------+----------------+------+-----+------------------+-------------------+----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode              | rstdev         |
+--------------------+----------------+------+-----+------------------+-------------------+----------------+
| AvroLoaderBench    | bench_load_10k | 1    | 3   | 93.195mb +0.00%  | 662.256ms -19.40% | ±0.60% -18.40% |
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 47.109mb +0.00%  | 69.659ms -15.00%  | ±0.79% +5.51%  |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 88.543mb +0.00%  | 67.687ms -24.08%  | ±0.61% -57.29% |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 281.593mb +1.87% | 1.443s -23.21%    | ±0.33% -33.54% |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 16.530mb +0.00%  | 35.584ms -16.56%  | ±1.05% -40.01% |
+--------------------+----------------+------+-----+------------------+-------------------+----------------+

Building Blocks

+-------------------------+----------------------------+------+-----+-----------------+-------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak        | mode              | rstdev          |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+-----------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 60.656mb +0.00% | 3.442ms -32.87%   | ±1.39% -18.32%  |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 80.448mb +0.00% | 176.215ms -16.81% | ±0.41% -83.28%  |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 58.974mb +0.00% | 17.819ms -21.27%  | ±0.04% -98.74%  |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 59.794mb +0.00% | 2.783ms -26.88%   | ±1.25% -41.38%  |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 59.794mb +0.00% | 2.708ms -26.87%   | ±3.41% +10.61%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 59.008mb +0.00% | 3.556ms -23.38%   | ±2.74% +13.92%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 59.537mb +0.00% | 17.577ms -32.84%  | ±0.80% +52.73%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 59.536mb +0.00% | 17.824ms -32.13%  | ±0.80% -60.99%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 57.608mb +0.00% | 2.294μs -8.46%    | ±2.08% +11.76%  |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 57.608mb +0.00% | 0.500μs 0.00%     | ±0.00% 0.00%    |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 65.841mb +0.00% | 13.804ms -18.23%  | ±1.84% +9.45%   |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 91.361mb +0.00% | 57.612ms -20.63%  | ±0.62% +35.53%  |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 60.057mb +0.00% | 3.080ms -22.35%   | ±1.56% +46.78%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 62.327mb +0.00% | 41.081ms -28.65%  | ±0.20% -55.07%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 62.158mb +0.00% | 6.912ms -28.46%   | ±1.30% -41.35%  |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 57.608mb +0.00% | 45.922ms -26.19%  | ±0.20% -69.36%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 57.608mb +0.00% | 46.035ms -25.41%  | ±0.56% -36.73%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 57.608mb +0.00% | 46.424ms -25.14%  | ±0.38% -41.70%  |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 59.882mb +0.00% | 9.875ms -12.56%   | ±1.41% +161.22% |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 57.607mb +0.00% | 36.320ms -20.46%  | ±0.50% +42.09%  |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 57.608mb +0.00% | 20.779μs -19.19%  | ±0.60% +63.61%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 57.608mb +0.00% | 26.688μs -16.55%  | ±0.98% +151.22% |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 80.449mb +0.00% | 178.968ms -18.88% | ±0.65% -62.74%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 91.727mb +0.00% | 142.847ms -17.06% | ±0.74% -11.83%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 47.590mb +0.00% | 72.715ms -16.93%  | ±0.19% -80.99%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 12.394mb -0.09% | 17.527ms -14.35%  | ±0.11% -92.87%  |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+-----------------+