Added BatchSizeOptimization

[norberttech]

Change Log

Added

• BatchSizeOptimization
• httpClient option to Meilisearch loader configuration

Fixed

Changed

• Optimizer can be now configured through ConfigBuilder
• Execution Plan Processors can be now configured through ConfigBuilder

Removed

Deprecated

Security

---

Description

Closes: #738

[norberttech]

@stloyd I noticed that Meilisearch has two methods, bulk_index, and bulk_update, but the update does not seem to be used in any way. Do you think it should be removed?

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+-------------------+------+-----+------------------+------------------+----------------+
| benchmark             | subject           | revs | its | mem_peak         | mode             | rstdev         |
+-----------------------+-------------------+------+-----+------------------+------------------+----------------+
| AvroExtractorBench    | bench_extract_10k | 1    | 3   | 35.274mb -0.03%  | 444.207ms +0.18% | ±0.09% -64.26% |
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 4.838mb +1.63%   | 340.241ms +0.22% | ±0.40% -3.20%  |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 5.008mb +0.33%   | 707.812ms +0.50% | ±0.09% -59.38% |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 239.705mb +0.01% | 1.008s -0.40%    | ±0.22% -51.98% |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 4.832mb +0.39%   | 23.314ms +1.54%  | ±1.09% +32.74% |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 4.832mb +0.39%   | 558.392ms -0.52% | ±0.37% -88.07% |
+-----------------------+-------------------+------+-----+------------------+------------------+----------------+

Transformers

+-----------------------------+--------------------------+------+-----+-----------------+-----------------+---------------+
| benchmark                   | subject                  | revs | its | mem_peak        | mode            | rstdev        |
+-----------------------------+--------------------------+------+-----+-----------------+-----------------+---------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 87.098mb +0.01% | 69.667ms -0.61% | ±0.63% +0.66% |
+-----------------------------+--------------------------+------+-----+-----------------+-----------------+---------------+

Loaders

+--------------------+----------------+------+-----+------------------+------------------+----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode             | rstdev         |
+--------------------+----------------+------+-----+------------------+------------------+----------------+
| AvroLoaderBench    | bench_load_10k | 1    | 3   | 95.281mb +0.02%  | 701.595ms -3.28% | ±0.82% -40.13% |
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 46.114mb +0.04%  | 70.183ms +0.08%  | ±1.14% -7.07%  |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 90.559mb +0.02%  | 79.761ms -2.60%  | ±0.32% -80.61% |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 292.401mb +0.01% | 2.136s -1.79%    | ±0.28% -37.14% |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 16.608mb +0.10%  | 36.980ms +0.98%  | ±0.71% -32.59% |
+--------------------+----------------+------+-----+------------------+------------------+----------------+

Building Blocks

+-------------------------+----------------------------+------+-----+-----------------+------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak        | mode             | rstdev          |
+-------------------------+----------------------------+------+-----+-----------------+------------------+-----------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 60.708mb +0.00% | 4.010ms -2.59%   | ±2.25% -13.12%  |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 80.500mb +0.00% | 175.837ms -0.72% | ±1.68% +224.43% |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 59.026mb +0.00% | 17.847ms -0.45%  | ±0.77% +10.90%  |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 59.846mb +0.00% | 2.901ms -10.32%  | ±2.06% -29.09%  |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 59.846mb +0.00% | 2.861ms -14.11%  | ±0.78% -63.82%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 59.060mb +0.00% | 4.105ms -6.53%   | ±3.64% +100.23% |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 59.589mb +0.00% | 25.278ms -2.02%  | ±0.81% -49.76%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 59.588mb +0.00% | 25.143ms -0.88%  | ±0.36% -65.07%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 57.659mb +0.00% | 2.500μs -0.24%   | ±0.00% -100.00% |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 57.659mb +0.00% | 0.500μs -16.67%  | ±0.00% 0.00%    |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 65.893mb +0.00% | 13.666ms -1.61%  | ±1.07% +51.99%  |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 91.412mb +0.00% | 63.347ms -0.35%  | ±0.12% -82.85%  |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 60.109mb +0.00% | 3.118ms -12.05%  | ±0.65% -53.35%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 62.379mb +0.00% | 49.746ms -2.41%  | ±0.44% -44.19%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 62.210mb +0.00% | 8.001ms -2.10%   | ±1.44% +9.50%   |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 57.659mb +0.00% | 50.431ms -0.95%  | ±0.43% -24.36%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 57.659mb +0.00% | 49.379ms -1.80%  | ±0.48% +34.33%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 57.659mb +0.00% | 51.578ms +0.76%  | ±1.57% +86.39%  |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 59.934mb +0.00% | 9.364ms -0.62%   | ±0.50% +228.57% |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 57.659mb +0.00% | 37.685ms -0.36%  | ±0.25% -55.75%  |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 57.659mb +0.00% | 21.136μs -4.22%  | ±1.33% -29.13%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 57.659mb +0.00% | 26.104μs -4.35%  | ±1.53% -22.58%  |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 80.500mb +0.00% | 179.595ms -1.13% | ±0.46% -53.77%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 93.715mb +0.00% | 155.361ms -0.24% | ±0.44% +16.19%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 48.630mb +0.01% | 79.077ms -1.75%  | ±0.19% -60.57%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 12.653mb -0.02% | 18.607ms -3.64%  | ±0.31% -67.82%  |
+-------------------------+----------------------------+------+-----+-----------------+------------------+-----------------+

[stloyd]

@norberttech I think that it's leftover as the implementation started to be similar to ElasticSearch, but MeiliSearch has "better" handling of updates for multiple documents as it uses the same endpoint as for new ones.

I would leave it and open a new issue to validate if we want to have:

• add new and update existing with given IDs (current),
• or add new and replace existing with given IDs,