Improve performance of TypeDetector

[stloyd]

Change Log

Added

Fixed

Changed

• Improve performance of `TypeDetector`

Removed

Deprecated

Security

---

Description

Result based on script:

Details

<?php

declare(strict_types=1);

include __DIR__ . '/../vendor/autoload.php';
include __DIR__ . '/../tools/blackfire/vendor/autoload.php';

$blackfire = new \Blackfire\Client();
$probe = $blackfire->createProbe(null, false);

// Disable profiling for the irrelevant parts of the code
$probe->disable();

$callback = static fn (int $i) : array => [
    'order_id' => '2d76fb83-c1a7-4b6e-9d68-46258af783b4',
    'created_at' => (new \DateTimeImmutable('-10 day ago'))->format(\DateTimeInterface::RFC3339),
    'updated_at' => (new \DateTimeImmutable('+1 week'))->format(\DateTimeInterface::RFC3339),
    'active' => true,
    'total_price' => 123.45,
    'discount' => 5.5,
    'customer' => [
        'name' => 'firstName',
        'last_name' => 'lastName',
        'email' => '[email protected]',
    ],
    'address' => [
        'street' => 'streetAddress',
        'city' => 'city',
        'zip' => '12-345',
        'country' => 'country',
        'location' => [
            'lat' => 66.6,
            'lng' => 33.3,
        ],
    ],
    'notes' => "Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.",
];

$data = \array_map($callback, \range(1, 5000));

// Actual profiling starts here
$probe->enable();

(new \Flow\ETL\PHP\Type\TypeDetector())->detectType($data);

// Profiling ends here
$profile = $blackfire->endProbe($probe);
var_dump($profile->getUrl());

[github-actions]

Flow PHP - Benchmarks

^{_{Results of the benchmarks from this PR are compared with the results from 1.x branch.}}

Extractors

+-----------------------+-------------------+------+-----+------------------+------------------+-----------------+
| benchmark             | subject           | revs | its | mem_peak         | mode             | rstdev          |
+-----------------------+-------------------+------+-----+------------------+------------------+-----------------+
| AvroExtractorBench    | bench_extract_10k | 1    | 3   | 34.747mb -0.01%  | 1.116s -2.86%    | ±0.43% +62.96%  |
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 4.605mb -0.04%   | 296.127ms -2.68% | ±0.85% -25.81%  |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 4.771mb -0.05%   | 1.350s -1.90%    | ±0.90% -57.56%  |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 239.480mb -0.00% | 1.544s -3.57%    | ±0.95% +148.28% |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 4.560mb +0.01%   | 23.491ms -5.97%  | ±0.45% -75.74%  |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 4.560mb +0.01%   | 405.331ms +0.31% | ±0.45% -56.71%  |
+-----------------------+-------------------+------+-----+------------------+------------------+-----------------+

Transformers

+-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+
| benchmark                   | subject                  | revs | its | mem_peak         | mode            | rstdev        |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 110.247mb -0.00% | 61.374ms -4.37% | ±0.81% +3.90% |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+

Loaders

+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode             | rstdev          |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| AvroLoaderBench    | bench_load_10k | 1    | 3   | 94.727mb -0.00%  | 436.637ms -2.40% | ±0.60% +15.40%  |
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 54.711mb -0.00%  | 70.300ms -1.62%  | ±1.73% +354.53% |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 105.309mb -0.00% | 54.068ms -3.40%  | ±0.18% -25.23%  |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 320.788mb -0.00% | 1.468s -1.80%    | ±0.26% -56.77%  |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 17.589mb -0.01%  | 41.622ms -0.83%  | ±0.15% -76.29%  |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+

Building Blocks

+-------------------------+----------------------------+------+-----+------------------+------------------+------------------+
| benchmark               | subject                    | revs | its | mem_peak         | mode             | rstdev           |
+-------------------------+----------------------------+------+-----+------------------+------------------+------------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 76.293mb -0.00%  | 2.329ms -11.88%  | ±2.71% +337.03%  |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 96.084mb -0.00%  | 179.556ms -2.58% | ±0.56% +58.01%   |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 74.610mb -0.00%  | 18.309ms -1.44%  | ±0.83% -40.75%   |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 75.432mb -0.00%  | 1.819ms -8.99%   | ±0.79% -68.94%   |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 75.432mb -0.00%  | 1.696ms -16.53%  | ±3.49% +317.53%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 74.645mb -0.00%  | 2.805ms -1.39%   | ±1.52% +119.14%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 75.174mb -0.00%  | 14.468ms -0.29%  | ±1.03% +255.20%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 75.174mb -0.00%  | 14.438ms -0.71%  | ±1.22% +8.87%    |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 73.077mb -0.00%  | 1.794μs -0.34%   | ±2.67% +0.00%    |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 73.077mb -0.00%  | 0.400μs +100.00% | ±0.00% 0.00%     |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 86.634mb -0.00%  | 12.134ms -7.48%  | ±0.28% -82.54%   |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 115.993mb -0.00% | 60.840ms -9.72%  | ±2.87% +25.03%   |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 75.695mb -0.00%  | 1.881ms -0.82%   | ±3.22% -0.69%    |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 77.963mb -0.00%  | 32.360ms -3.44%  | ±0.84% -75.82%   |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 77.795mb -0.00%  | 4.558ms -2.78%   | ±1.67% -37.76%   |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 73.220mb -0.00%  | 38.206ms -1.36%  | ±1.50% +131.54%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 73.221mb -0.00%  | 38.424ms -1.51%  | ±0.53% +1192.85% |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 73.220mb -0.00%  | 38.701ms -1.23%  | ±0.69% +27.95%   |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 75.520mb -0.00%  | 7.349ms +0.77%   | ±1.56% +104.96%  |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 73.077mb -0.00%  | 28.609ms +0.87%  | ±0.47% -50.21%   |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 73.077mb -0.00%  | 13.043μs -2.02%  | ±2.51% +255.32%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 73.077mb -0.00%  | 15.706μs -2.46%  | ±0.30% -86.61%   |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 96.085mb -0.00%  | 182.275ms -1.46% | ±1.68% +285.42%  |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 98.214mb -0.00%  | 932.880ms -3.43% | ±0.44% -30.09%   |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 21.905mb -0.01%  | 184.142ms -3.44% | ±0.28% -63.15%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 115.837mb -0.00% | 734.863ms -5.36% | ±0.74% -68.83%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 59.556mb -0.00%  | 372.187ms -4.89% | ±0.40% -72.37%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 14.678mb -0.01%  | 76.017ms -1.88%  | ±0.32% -88.36%   |
+-------------------------+----------------------------+------+-----+------------------+------------------+------------------+