Skip to content

Conversation

@norberttech
Copy link
Member

@norberttech norberttech commented Oct 13, 2024

Change Log

Added

  • file:read - cli command
  • file:schema- cli command
  • file:rows:command- cli command

Fixed

Changed

  • Unified options/arguments handling across all commands
  • run cli command was renamed into pipeline:run with alias run

Removed

Deprecated

Security


Description

Examples:

$ flow read azure-blob://orders.parquet --config=./.flow.config.php --parquet-columns=order_id 
+----------------------+
|             order_id |
+----------------------+
| e13d7098-5a78-3389-9 |
| 947df050-3abb-3f5a-9 |
| 6315f9e2-86bf-3321-a |
| 4cccb632-fade-34e2-8 |
| 82384f8c-9adb-38be-9 |
| e3fcf736-0f8c-3d97-b |
| b987a49a-b4c5-37de-b |
| 663523a9-713b-3354-8 |
| 6259fa2c-ec68-36a9-8 |
| f7153c83-34b6-3769-9 |
| 966b91b5-e252-3787-9 |
| 607e5afa-3783-39ce-8 |
| 05476f35-2efc-3708-a |
| 71f9884e-3d07-3f69-8 |
| de23917a-15bd-33cc-9 |
| 10a8b132-11ca-3288-9 |
| 384f222d-fb15-33ba-b |
| c36018e3-4368-3660-8 |
| 371fa03b-970e-3ad3-9 |
+----------------------+
19 rows

$ flow schema azure-blob://orders.parquet --config=.flow.config.php --output-table             
+------------+--------+----------+-------------+----------+
|       name |   type | nullable | scalar_type | metadata |
+------------+--------+----------+-------------+----------+
|   order_id |   uuid |    false |             |       [] |
| created_at | scalar |    false |      string |       [] |
| updated_at | scalar |    false |      string |       [] |
|   discount | scalar |    false |      string |       [] |
|    address |   json |    false |             |       [] |
|      notes |   json |    false |             |       [] |
|      items |   json |    false |             |       [] |
+------------+--------+----------+-------------+----------+
7 rows

@github-actions
Copy link
Contributor

github-actions bot commented Oct 13, 2024

Flow PHP - Benchmarks

Results of the benchmarks from this PR are compared with the results from 1.x branch.

Extractors
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| benchmark             | subject           | revs | its | mem_peak        | mode             | rstdev          |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
| CSVExtractorBench     | bench_extract_10k | 1    | 3   | 4.602mb +0.32%  | 516.240ms +0.02% | ±0.15% -58.25%  |
| JsonExtractorBench    | bench_extract_10k | 1    | 3   | 4.690mb +0.32%  | 1.085s -0.88%    | ±1.55% +96.65%  |
| ParquetExtractorBench | bench_extract_10k | 1    | 3   | 29.145mb +0.05% | 441.986ms -1.61% | ±1.44% +145.41% |
| TextExtractorBench    | bench_extract_10k | 1    | 3   | 4.330mb +0.34%  | 33.460ms -0.77%  | ±0.49% -41.02%  |
| XmlExtractorBench     | bench_extract_10k | 1    | 3   | 4.312mb +0.34%  | 649.049ms -0.46% | ±0.35% -76.38%  |
+-----------------------+-------------------+------+-----+-----------------+------------------+-----------------+
Transformers
+-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+
| benchmark                   | subject                  | revs | its | mem_peak         | mode            | rstdev          |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+
| RenameEntryTransformerBench | bench_transform_10k_rows | 1    | 3   | 116.606mb +0.01% | 61.789ms -0.31% | ±3.01% +183.03% |
+-----------------------------+--------------------------+------+-----+------------------+-----------------+-----------------+
Loaders
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| benchmark          | subject        | revs | its | mem_peak         | mode             | rstdev          |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
| CSVLoaderBench     | bench_load_10k | 1    | 3   | 54.799mb +0.03%  | 142.587ms +0.16% | ±0.75% +317.78% |
| JsonLoaderBench    | bench_load_10k | 1    | 3   | 90.382mb +0.02%  | 118.108ms +0.07% | ±0.61% +22.64%  |
| ParquetLoaderBench | bench_load_10k | 1    | 3   | 124.436mb +0.01% | 1.255s -2.16%    | ±0.59% -1.14%   |
| TextLoaderBench    | bench_load_10k | 1    | 3   | 17.521mb +0.08%  | 45.029ms +0.05%  | ±0.11% -96.54%  |
+--------------------+----------------+------+-----+------------------+------------------+-----------------+
Building Blocks
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| benchmark               | subject                    | revs | its | mem_peak         | mode             | rstdev          |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+
| RowsBench               | bench_chunk_10_on_10k      | 2    | 3   | 87.355mb +0.02%  | 3.502ms -13.78%  | ±2.93% -7.79%   |
| RowsBench               | bench_diff_left_1k_on_10k  | 2    | 3   | 102.958mb +0.01% | 186.124ms -2.40% | ±0.20% -50.50%  |
| RowsBench               | bench_diff_right_1k_on_10k | 2    | 3   | 85.678mb +0.02%  | 18.931ms -2.03%  | ±1.35% +93.27%  |
| RowsBench               | bench_drop_1k_on_10k       | 2    | 3   | 88.595mb +0.02%  | 1.942ms -1.78%   | ±3.58% -0.17%   |
| RowsBench               | bench_drop_right_1k_on_10k | 2    | 3   | 88.595mb +0.02%  | 2.160ms +8.91%   | ±3.03% +22.96%  |
| RowsBench               | bench_entries_on_10k       | 2    | 3   | 85.707mb +0.02%  | 3.056ms -3.15%   | ±1.01% -46.80%  |
| RowsBench               | bench_filter_on_10k        | 2    | 3   | 86.236mb +0.02%  | 15.842ms +3.15%  | ±3.15% +146.45% |
| RowsBench               | bench_find_on_10k          | 2    | 3   | 86.236mb +0.02%  | 15.946ms +1.06%  | ±1.80% -44.64%  |
| RowsBench               | bench_find_one_on_10k      | 10   | 3   | 84.139mb +0.02%  | 1.794μs -5.28%   | ±2.67% +5.66%   |
| RowsBench               | bench_first_on_10k         | 10   | 3   | 84.139mb +0.02%  | 0.400μs 0.00%    | ±0.00% 0.00%    |
| RowsBench               | bench_flat_map_on_1k       | 2    | 3   | 93.490mb +0.01%  | 12.763ms -2.40%  | ±0.64% -22.39%  |
| RowsBench               | bench_map_on_10k           | 2    | 3   | 122.861mb +0.01% | 63.296ms -1.22%  | ±0.35% -47.52%  |
| RowsBench               | bench_merge_1k_on_10k      | 2    | 3   | 86.755mb +0.02%  | 1.632ms -7.17%   | ±2.75% +285.31% |
| RowsBench               | bench_partition_by_on_10k  | 2    | 3   | 90.107mb +0.02%  | 66.383ms +0.44%  | ±0.87% -30.29%  |
| RowsBench               | bench_remove_on_10k        | 2    | 3   | 88.857mb +0.02%  | 4.342ms -2.74%   | ±0.22% -92.66%  |
| RowsBench               | bench_sort_asc_on_1k       | 2    | 3   | 84.289mb +0.02%  | 40.664ms -0.39%  | ±2.15% +115.56% |
| RowsBench               | bench_sort_by_on_1k        | 2    | 3   | 84.289mb +0.02%  | 39.562ms -8.24%  | ±0.68% -28.72%  |
| RowsBench               | bench_sort_desc_on_1k      | 2    | 3   | 84.289mb +0.02%  | 39.998ms -8.37%  | ±1.30% +116.13% |
| RowsBench               | bench_sort_entries_on_1k   | 2    | 3   | 86.581mb +0.02%  | 7.467ms -4.97%   | ±1.07% -44.39%  |
| RowsBench               | bench_sort_on_1k           | 2    | 3   | 84.139mb +0.02%  | 29.615ms -4.00%  | ±2.84% +449.75% |
| RowsBench               | bench_take_1k_on_10k       | 10   | 3   | 84.139mb +0.02%  | 13.336μs -5.95%  | ±2.10% +75.86%  |
| RowsBench               | bench_take_right_1k_on_10k | 10   | 3   | 84.139mb +0.02%  | 16.099μs -4.81%  | ±1.52% +173.79% |
| RowsBench               | bench_unique_on_1k         | 2    | 3   | 102.959mb +0.01% | 188.833ms -2.39% | ±1.08% +191.03% |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 53.181mb +0.03%  | 396.524ms -0.91% | ±0.25% +148.55% |
| TypeDetectorBench       | bench_type_detector        | 1    | 3   | 13.447mb +0.10%  | 80.274ms -1.30%  | ±0.27% -39.20%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 107.443mb +0.01% | 488.014ms -0.68% | ±0.42% +50.67%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 55.801mb +0.02%  | 241.721ms -0.37% | ±2.08% +12.54%  |
| NativeEntryFactoryBench | bench_entry_factory        | 1    | 3   | 14.639mb +0.10%  | 52.628ms -2.32%  | ±1.19% +309.81% |
+-------------------------+----------------------------+------+-----+------------------+------------------+-----------------+

@norberttech norberttech merged commit c269cc5 into flow-php:1.x Oct 14, 2024
@norberttech norberttech deleted the feature/cli-commands branch December 5, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant