Fixed massive performance bottleneck in parquet library

[norberttech]

Resolves: #xxx

Change Log

---

Added

• trigger gc_collect_cycles before starting pipeline execution hr time
• schema_to_php dsl function
• schema_to_ascii dsl function

Fixed

• performance bottleneck in parquet library
• don't include pcov in nix shell when blackfire is enabled

Changed

Removed

Deprecated

Security

Before writing 100k rows to a file was taking ~65sec.
Now it will take ~5sec

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+------------------------+------+-----+-----------------+------------------+-----------------+
| benchmark             | subject                | revs | its | mem_peak        | mode             | rstdev          |
+-----------------------+------------------------+------+-----+-----------------+------------------+-----------------+
| CSVExtractorBench     | bench_extract_10k      | 1    | 3   | 4.824mb +0.08%  | 437.104ms -2.51% | ±0.76% -58.55%  |
| ExcelExtractorBench   | bench_extract_10k_ods  | 1    | 3   | 65.535mb +0.01% | 1.061s -0.24%    | ±1.87% +178.24% |
| ExcelExtractorBench   | bench_extract_10k_xlsx | 1    | 3   | 67.581mb +0.01% | 1.681s +0.09%    | ±0.80% -43.15%  |
| JsonExtractorBench    | bench_extract_10k      | 1    | 3   | 5.320mb +2.14%  | 1.153s -1.12%    | ±0.63% +167.82% |
| ParquetExtractorBench | bench_extract_10k      | 1    | 3   | 86.394mb +0.01% | 898.188ms -3.08% | ±1.52% -31.16%  |
| TextExtractorBench    | bench_extract_10k      | 1    | 3   | 4.564mb +0.09%  | 41.865ms -0.50%  | ±0.60% +72.85%  |
| XmlExtractorBench     | bench_extract_10k      | 1    | 3   | 4.549mb +0.09%  | 601.843ms -0.28% | ±0.65% +0.11%   |
+-----------------------+------------------------+------+-----+-----------------+------------------+-----------------+

Transformers

+---------------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| benchmark                       | subject                  | revs | its | mem_peak         | mode            | rstdev         |
+---------------------------------+--------------------------+------+-----+------------------+-----------------+----------------+
| RenameEachEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 18.559mb +0.02%  | 71.845ms +0.99% | ±0.93% +14.52% |
| RenameEntryTransformerBench     | bench_transform_10k_rows | 1    | 3   | 123.298mb +0.00% | 65.525ms -0.16% | ±0.80% -57.17% |
+---------------------------------+--------------------------+------+-----+------------------+-----------------+----------------+

Loaders

+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode             | rstdev          |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 62.486mb +0.01%  | 86.361ms +0.14%  | ±0.40% +535.35% |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 80.581mb +0.00%  | 102.911ms -2.10% | ±0.72% -12.91%  |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 166.292mb +0.00% | 1.992s -90.36%   | ±0.99% +665.09% |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 17.865mb +0.02%  | 30.139ms -0.40%  | ±0.57% -27.67%  |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+

Building Blocks

+-------------------+----------------------------+------+-----+------------------+------------------+------------------+
| benchmark         | subject                    | revs | its | mem_peak         | mode             | rstdev           |
+-------------------+----------------------------+------+-----+------------------+------------------+------------------+
| TypeDetectorBench | bench_type_detector        | 1    | 3   | 42.509mb +0.01%  | 404.645ms -0.03% | ±0.67% +57.35%   |
| TypeDetectorBench | bench_type_detector        | 1    | 3   | 11.566mb +0.03%  | 82.158ms +0.75%  | ±0.49% -44.38%   |
| EntryFactoryBench | bench_entry_factory        | 1    | 3   | 105.978mb +0.00% | 650.376ms -0.12% | ±1.75% +278.18%  |
| EntryFactoryBench | bench_entry_factory        | 1    | 3   | 55.253mb +0.01%  | 322.731ms -1.87% | ±1.22% +53.05%   |
| EntryFactoryBench | bench_entry_factory        | 1    | 3   | 14.839mb +0.03%  | 68.862ms +0.30%  | ±0.60% -0.63%    |
| RowsBench         | bench_chunk_10_on_10k      | 2    | 3   | 93.450mb +0.00%  | 3.235ms +2.05%   | ±1.80% +1225.44% |
| RowsBench         | bench_diff_left_1k_on_10k  | 2    | 3   | 110.820mb +0.00% | 240.363ms +1.19% | ±0.48% +191.63%  |
| RowsBench         | bench_diff_right_1k_on_10k | 2    | 3   | 93.540mb +0.00%  | 23.909ms +1.24%  | ±0.15% +98.44%   |
| RowsBench         | bench_drop_1k_on_10k       | 2    | 3   | 94.324mb +0.00%  | 1.326ms +4.94%   | ±3.05% +288.41%  |
| RowsBench         | bench_drop_right_1k_on_10k | 2    | 3   | 94.324mb +0.00%  | 1.406ms +7.58%   | ±3.36% +15.70%   |
| RowsBench         | bench_entries_on_10k       | 2    | 3   | 92.485mb +0.00%  | 3.243ms -0.82%   | ±1.53% -44.70%   |
| RowsBench         | bench_filter_on_10k        | 2    | 3   | 93.014mb +0.00%  | 16.151ms +5.98%  | ±0.89% +180.89%  |
| RowsBench         | bench_find_on_10k          | 2    | 3   | 93.014mb +0.00%  | 15.440ms +2.37%  | ±1.18% +629.20%  |
| RowsBench         | bench_find_one_on_10k      | 10   | 3   | 91.703mb +0.00%  | 1.806μs 0.00%    | ±2.57% 0.00%     |
| RowsBench         | bench_first_on_10k         | 10   | 3   | 91.703mb +0.00%  | 0.400μs 0.00%    | ±0.00% 0.00%     |
| RowsBench         | bench_flat_map_on_1k       | 2    | 3   | 100.763mb +0.00% | 14.178ms -0.78%  | ±0.27% -72.32%   |
| RowsBench         | bench_map_on_10k           | 2    | 3   | 130.190mb +0.00% | 65.423ms -0.40%  | ±0.85% -49.79%   |
| RowsBench         | bench_merge_1k_on_10k      | 2    | 3   | 93.534mb +0.00%  | 1.145ms +4.47%   | ±3.55% +900.82%  |
| RowsBench         | bench_partition_by_on_10k  | 2    | 3   | 96.903mb +0.00%  | 61.234ms +1.56%  | ±0.05% -95.75%   |
| RowsBench         | bench_remove_on_10k        | 2    | 3   | 94.587mb +0.00%  | 3.431ms +0.81%   | ±0.58% +25.14%   |
| RowsBench         | bench_sort_asc_on_1k       | 2    | 3   | 92.065mb +0.00%  | 39.852ms +0.18%  | ±2.05% +48.18%   |
| RowsBench         | bench_sort_by_on_1k        | 2    | 3   | 92.065mb +0.00%  | 40.301ms +2.30%  | ±3.03% +441.99%  |
| RowsBench         | bench_sort_desc_on_1k      | 2    | 3   | 92.065mb +0.00%  | 40.759ms +2.48%  | ±0.93% -10.40%   |
| RowsBench         | bench_sort_entries_on_1k   | 2    | 3   | 94.146mb +0.00%  | 8.215ms +0.83%   | ±0.66% -55.49%   |
| RowsBench         | bench_sort_on_1k           | 2    | 3   | 91.896mb +0.00%  | 29.632ms -0.97%  | ±0.54% -54.22%   |
| RowsBench         | bench_take_1k_on_10k       | 10   | 3   | 91.703mb +0.00%  | 13.643μs +0.63%  | ±2.40% +11.55%   |
| RowsBench         | bench_take_right_1k_on_10k | 10   | 3   | 91.703mb +0.00%  | 15.400μs -9.42%  | ±0.00% -100.00%  |
| RowsBench         | bench_unique_on_1k         | 2    | 3   | 110.820mb +0.00% | 241.366ms -0.14% | ±0.63% -74.95%   |
+-------------------+----------------------------+------+-----+------------------+------------------+------------------+

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.23%. Comparing base (c03c21c) to head (8a0acc3).
Report is 2 commits behind head on 1.x.

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##              1.x    #1735      +/-   ##
==========================================
- Coverage   81.23%   81.23%   -0.01%     
==========================================
  Files         715      715              
  Lines       19872    19873       +1     
==========================================
- Hits        16144    16143       -1     
- Misses       3728     3730       +2     

Components	Coverage Δ
etl	88.33% <100.00%> (+<0.01%)	⬆️
cli	85.46% <ø> (ø)
lib-array-dot	94.56% <ø> (ø)
lib-azure-sdk	61.35% <ø> (ø)
lib-doctrine-dbal-bulk	93.88% <ø> (ø)
lib-filesystem	78.02% <ø> (ø)
lib-types	53.57% <ø> (ø)
lib-parquet	84.14% <100.00%> (ø)
lib-parquet-viewer	83.11% <ø> (ø)
lib-snappy	90.69% <ø> (-0.94%)	⬇️
bridge-filesystem-async-aws	90.38% <ø> (ø)
bridge-filesystem-azure	89.92% <ø> (ø)
bridge-monolog-http	97.04% <ø> (ø)
symfony-http-foundation	74.41% <ø> (ø)
adapter-chartjs	86.70% <ø> (ø)
adapter-csv	87.74% <ø> (ø)
adapter-doctrine	89.89% <ø> (ø)
adapter-elasticsearch	97.23% <ø> (ø)
adapter-google-sheet	83.87% <ø> (ø)
adapter-http	58.10% <ø> (ø)
adapter-json	87.98% <ø> (ø)
adapter-logger	53.84% <ø> (ø)
adapter-meilisearch	97.95% <ø> (ø)
adapter-parquet	78.64% <ø> (ø)
adapter-text	84.44% <ø> (ø)
adapter-xml	82.73% <ø> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.