Improve adapters benchmark stability

[stloyd]

Change Log

Added

Fixed

Changed

• Improve adapters benchmark stability

Removed

Deprecated

Security

---

Description

Added retry-threshold back to settings, which allows getting better rstdev values as it re-tries benchmark if the value is over the given value.

Moved the setup of files out of benchmark methods, which should make those more reliable.

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+-------------------+------+-----+------------------+-------------------+------------------+
| benchmark             | subject           | revs | its | mem_peak         | mode              | rstdev           |
+-----------------------+-------------------+------+-----+------------------+-------------------+------------------+
| AvroExtractorBench    | bench_extract_10k | 1    | 3   | 44.121mb -0.00%  | 553.080ms +32.68% | ±0.26% -72.38%   |
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 14.006mb -0.00%  | 444.002ms +33.30% | ±1.14% +134.17%  |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 18.668mb -0.00%  | 852.333ms +31.02% | ±2.03% +555.64%  |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 237.797mb -0.01% | 1.210s +35.22%    | ±1.20% +2271.43% |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 7.275mb -0.01%   | 22.658ms +55.01%  | ±2.00% +93.05%   |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 7.623mb -0.01%   | 743.007ms +34.58% | ±1.18% +259.76%  |
+-----------------------+-------------------+------+-----+------------------+-------------------+------------------+

Transformers

+-----------------------------+--------------------------+------+-----+-----------------+------------------+-----------------+
| benchmark                   | subject                  | revs | its | mem_peak        | mode             | rstdev          |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+-----------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 87.061mb +0.00% | 88.547ms +43.65% | ±3.37% +776.97% |
+-----------------------------+--------------------------+------+-----+-----------------+------------------+-----------------+

Loaders

+--------------------+----------------+------+-----+-------------------+-------------------+------------------+
| benchmark          | subject        | revs | its | mem_peak          | mode              | rstdev           |
+--------------------+----------------+------+-----+-------------------+-------------------+------------------+
| AvroLoaderBench    | bench_load_10k | 1    | 3   | 219.955mb -57.62% | 900.229ms +36.89% | ±0.75% +915.99%  |
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 47.139mb -0.00%   | 95.902ms +38.43%  | ±2.82% +934.10%  |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 88.578mb -0.01%   | 90.128ms +43.89%  | ±1.71% -44.51%   |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 281.205mb -0.00%  | 1.256s +32.16%    | ±2.18% +3303.59% |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 16.565mb -0.03%   | 44.522ms +24.24%  | ±2.42% +379.39%  |
+--------------------+----------------+------+-----+-------------------+-------------------+------------------+

Building Blocks

+-------------------------+----------------------------+------+-----+-----------------+-------------------+------------------+
| benchmark               | subject                    | revs | its | mem_peak        | mode              | rstdev           |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+------------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 60.685mb +0.00% | 5.615ms +79.68%   | ±3.28% +38.89%   |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 80.475mb +0.00% | 226.229ms +27.67% | ±0.97% +81.01%   |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 59.001mb +0.00% | 21.890ms +23.24%  | ±1.36% +1223.27% |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 59.824mb +0.00% | 3.559ms +40.63%   | ±2.89% -41.31%   |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 59.824mb +0.00% | 3.464ms +37.23%   | ±2.46% +373.73%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 59.037mb +0.00% | 4.810ms +33.14%   | ±0.07% -94.23%   |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 59.566mb +0.00% | 25.889ms +50.88%  | ±0.77% +40.98%   |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 59.565mb +0.00% | 26.764ms +54.69%  | ±2.27% +1300.26% |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 57.637mb +0.00% | 2.500μs +4.17%    | ±0.00% 0.00%     |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 57.637mb +0.00% | 0.500μs +25.06%   | ±0.00% -100.00%  |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 65.870mb +0.00% | 17.119ms +24.37%  | ±3.25% +83.76%   |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 91.390mb +0.00% | 77.339ms +36.34%  | ±2.85% +290.93%  |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 60.087mb +0.00% | 3.472ms +27.66%   | ±2.32% +35.51%   |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 62.355mb +0.00% | 51.190ms +34.99%  | ±2.40% +367.73%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 62.187mb +0.00% | 8.860ms +34.83%   | ±2.80% +135.47%  |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 57.637mb +0.00% | 60.972ms +33.85%  | ±1.19% +493.79%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 57.637mb +0.00% | 60.971ms +33.60%  | ±0.39% +156.08%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 57.637mb +0.00% | 66.252ms +44.43%  | ±2.26% +2376.48% |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 59.911mb +0.00% | 12.060ms +24.69%  | ±2.18% +64.03%   |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 57.636mb +0.00% | 44.753ms +24.74%  | ±1.83% +1127.44% |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 57.637mb +0.00% | 28.821μs +40.63%  | ±0.43% +88.24%   |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 57.637mb +0.00% | 37.460μs +44.60%  | ±0.76% +320.28%  |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 80.476mb +0.00% | 229.755ms +29.71% | ±1.67% +305.72%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 91.764mb -0.03% | 199.570ms +39.40% | ±3.66% +431.33%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 47.601mb +0.07% | 96.221ms +32.69%  | ±1.24% +512.81%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 12.417mb +0.00% | 23.446ms +36.74%  | ±2.05% +981.91%  |
+-------------------------+----------------------------+------+-----+-----------------+-------------------+------------------+