[fix](pipeline) Make all upstream tasks runnable if all tasks finishe…#41292
[fix](pipeline) Make all upstream tasks runnable if all tasks finishe…#41292Gabriel39 merged 13 commits intoapache:masterfrom
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
3 similar comments
|
run buildall |
|
run buildall |
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
|
TeamCity be ut coverage result: |
|
run buildall |
1 similar comment
|
run buildall |
|
TeamCity be ut coverage result: |
Gabriel39
force-pushed
the
fix_0925_1
branch
from
c6c8fc5 to
71361c6
Compare
apache#41292) Consider 3 pipelines in this fragment (... -> join -> shuffle) : pipeline 0 : `... -> local exchange sink` pipeline 1 : `... -> join build (INNER JOIN)` pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data stream sender ` Assume the JoinBuild returned 0 rows, join probe can finish directly once join build finished and do not need to wait for the `local exchange sink` finished. In this case, if pipeline 0 is blocked by a dependency for a long time, pipeline 2 should notify pipeline 0 to finish.
apache#41292) Consider 3 pipelines in this fragment (... -> join -> shuffle) : pipeline 0 : `... -> local exchange sink` pipeline 1 : `... -> join build (INNER JOIN)` pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data stream sender ` Assume the JoinBuild returned 0 rows, join probe can finish directly once join build finished and do not need to wait for the `local exchange sink` finished. In this case, if pipeline 0 is blocked by a dependency for a long time, pipeline 2 should notify pipeline 0 to finish.
apache#41292) Consider 3 pipelines in this fragment (... -> join -> shuffle) : pipeline 0 : `... -> local exchange sink` pipeline 1 : `... -> join build (INNER JOIN)` pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data stream sender ` Assume the JoinBuild returned 0 rows, join probe can finish directly once join build finished and do not need to wait for the `local exchange sink` finished. In this case, if pipeline 0 is blocked by a dependency for a long time, pipeline 2 should notify pipeline 0 to finish.
…ly (apache#41667) send ignored rf when hash join build closed early to avoid runtime filter sync/merge error Follow-up : apache#41292
…pache#41751) Sometimes eos is true, the finish dependency is still not ready. Therefore, we need to use wake_up_by_downstream to determine whether it was closed early. Otherwise, it will enter the normal rf build process in this case and generate related errors. Follow-up : apache#41292
…pache#41751) ## Proposed changes Sometimes eos is true, the finish dependency is still not ready. Therefore, we need to use wake_up_by_downstream to determine whether it was closed early. Otherwise, it will enter the normal rf build process in this case and generate related errors. Follow-up : apache#41292
apache#41292) Consider 3 pipelines in this fragment (... -> join -> shuffle) : pipeline 0 : `... -> local exchange sink` pipeline 1 : `... -> join build (INNER JOIN)` pipeline 2 : `local exchange source -> join probe (INNER JOIN) -> data stream sender ` Assume the JoinBuild returned 0 rows, join probe can finish directly once join build finished and do not need to wait for the `local exchange sink` finished. In this case, if pipeline 0 is blocked by a dependency for a long time, pipeline 2 should notify pipeline 0 to finish.
…ly (apache#41667) ## Proposed changes send ignored rf when hash join build closed early to avoid runtime filter sync/merge error Follow-up : apache#41292
…pache#41751) ## Proposed changes Sometimes eos is true, the finish dependency is still not ready. Therefore, we need to use wake_up_by_downstream to determine whether it was closed early. Otherwise, it will enter the normal rf build process in this case and generate related errors. Follow-up : apache#41292
4 participants
…d in this pipeline
Proposed changes
Consider 3 pipelines in this fragment (... -> join -> shuffle) :
pipeline 0 :
... -> local exchange sinkpipeline 1 :
... -> join build (INNER JOIN)pipeline 2 :
local exchange source -> join probe (INNER JOIN) -> data stream senderAssume the JoinBuild returned 0 rows, join probe can finish directly once join build finished and do not need to wait for the
local exchange sinkfinished. In this case, if pipeline 0 is blocked by a dependency for a long time, pipeline 2 should notify pipeline 0 to finish.