Skip to content

Conversation

@norberttech
Copy link
Member

@norberttech norberttech commented Sep 29, 2024

Change Log

Added

  • Added more output styles for to_output() loader

Fixed

Changed

Removed

Deprecated

Security


Description

Example:

df()
    ->read(from_parquet(__DIR__ . '/orders_with_products.parquet'))
    ->batchSize(1000)
    ->write(to_output(output: Output::rows_and_column_count))
    ->run();

Output:

php .scratchpad/flow/read.php
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9
Rows: 1000, Columns: 9

@github-actions
Copy link
Contributor

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| benchmark             | subject           | revs | its | mem_peak        | mode             | rstdev          |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 4.631mb +0.01%  | 521.060ms +0.79% | ±0.36% +267.85% |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 4.719mb +0.01%  | 1.093s +1.02%    | ±0.17% -42.81%  |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 29.175mb +0.00% | 436.013ms +0.16% | ±0.70% -49.44%  |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 4.360mb +0.01%  | 33.330ms -2.30%  | ±0.86% -43.12%  |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 4.341mb +0.01%  | 652.655ms -6.04% | ±0.72% -64.54%  |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
Transformers
+-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+
| benchmark                   | subject                  | revs | its | mem_peak         | mode            | rstdev        |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 116.701mb +0.00% | 60.429ms +0.76% | ±0.28% -5.43% |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+---------------+
Loaders
+--------------------+----------------+------+-----+------------------+------------------+------------------+
| benchmark          | subject        | revs | its | mem_peak         | mode             | rstdev           |
+--------------------+----------------+------+-----+------------------+------------------+------------------+
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 54.829mb +0.00%  | 119.763ms -2.67% | ±1.50% +1016.33% |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 90.412mb +0.00%  | 99.163ms -2.81%  | ±0.40% -70.77%   |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 124.532mb +0.00% | 1.231s -0.76%    | ±0.30% -73.34%   |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 17.551mb +0.00%  | 27.716ms -6.12%  | ±0.56% -56.77%   |
+--------------------+----------------+------+-----+------------------+------------------+------------------+
Building Blocks
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak         | mode             | rstdev          |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 53.271mb +0.00%  | 400.413ms +1.37% | ±0.73% -34.88%  |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 13.537mb +0.00%  | 80.791ms +0.87%  | ±0.30% -65.28%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 107.468mb +0.00% | 493.502ms +3.24% | ±0.79% +0.71%   |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 55.826mb +0.00%  | 243.685ms +0.75% | ±0.61% -29.71%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 14.664mb +0.00%  | 52.249ms -0.08%  | ±2.29% +311.38% |
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 87.445mb +0.00%  | 3.300ms -2.10%   | ±3.05% +108.35% |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 102.983mb +0.00% | 190.942ms +1.62% | ±0.45% -8.18%   |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 85.703mb +0.00%  | 19.133ms +1.61%  | ±0.54% -44.56%  |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 88.685mb +0.00%  | 1.859ms -1.84%   | ±1.17% -49.38%  |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 88.685mb +0.00%  | 1.823ms -3.56%   | ±0.67% -78.77%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 85.797mb +0.00%  | 2.746ms -2.85%   | ±1.60% +10.11%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 86.326mb +0.00%  | 17.879ms +7.93%  | ±0.94% -13.94%  |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 86.326mb +0.00%  | 18.118ms +6.89%  | ±2.38% +16.35%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 84.230mb +0.00%  | 1.694μs +5.47%   | ±2.83% -2.00%   |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 84.230mb +0.00%  | 0.300μs 0.00%    | ±0.00% 0.00%    |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 93.580mb +0.00%  | 12.576ms +0.14%  | ±0.70% -26.07%  |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 122.951mb +0.00% | 62.517ms +2.97%  | ±1.07% -20.69%  |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 86.846mb +0.00%  | 1.391ms -9.56%   | ±0.90% -72.69%  |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 90.136mb +0.00%  | 60.822ms +1.12%  | ±0.38% -83.40%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 88.947mb +0.00%  | 4.141ms -9.53%   | ±0.19% -93.44%  |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 84.314mb +0.00%  | 40.582ms +2.26%  | ±0.95% -21.57%  |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 84.314mb +0.00%  | 40.479ms +3.34%  | ±0.88% +45.05%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 84.314mb +0.00%  | 40.812ms +3.16%  | ±1.18% -16.47%  |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 86.671mb +0.00%  | 7.474ms -1.10%   | ±0.10% -92.20%  |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 84.230mb +0.00%  | 29.161ms +1.05%  | ±1.13% -12.38%  |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 84.230mb +0.00%  | 13.112μs -8.69%  | ±0.72% -80.20%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 84.230mb +0.00%  | 15.961μs -1.68%  | ±2.90% +128.47% |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 102.988mb +0.00% | 193.753ms +0.25% | ±0.36% -61.35%  |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+

@norberttech norberttech merged commit 4706273 into flow-php:1.x Sep 29, 2024
@norberttech norberttech deleted the feature/output-rows-count branch December 5, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant