Skip to content

Conversation

@norberttech
Copy link
Member

@norberttech norberttech commented Oct 1, 2022

Change Log

Added

  • Pipeline Execution Plan

Fixed

Changed

Removed

Deprecated

Security


Description

So the goal of the pipeline execution plan is to make it possible to process the logical plan and optimize/change it by the processor before actual execution.

Currently implementation mostly aims to simplify two things:

  1. Single place where SaveMode is interpreted
  2. Async processing of file to file fixes

This PR also unifies the behavior of SaveMode:

  • Append - only available in "thread_safe" flow context where the path is always treated as a folder and new files are just appended into it
  • Overwrite - path is always removed before processing the pipelines
  • IgnoreIfExists - loaders that are going to try to load into the existing location are skipped
  • ExceptionIfExists - if any loader is going to try to load into the existing location exception is thrown

Potential futuristic benefits of having an Execution Plan and Execution Processors:

  • Optimization of the pipelines (transformers mostly) even before execution
  • Calculation pipeline execution complexity
  • Early errors detection
  • Visualization of the execution plan

@github-actions
Copy link
Contributor

github-actions bot commented Oct 1, 2022

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 1, 2022

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR.

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Oct 1, 2022

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 1, 2022

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR.

@norberttech norberttech merged commit b7d46eb into flow-php:1.x Oct 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant