Support DeltaJoin on Flink 2.1

[swuferhong]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Support DeltaJoin on Flink2.1

Solution

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[xuyangzhong]

Hi, @swuferhong can I take this issue?

[swuferhong]

Hi, @swuferhong can I take this issue?

Done!

[polyzos]

In the previous version, the prefix lookup was introduced to support the delta-join. Is there more work that needs to be done, or is this ticket about supporting Flink 2.1 so we can leverage it?

[swuferhong]

In the previous version, the prefix lookup was introduced to support the delta-join. Is there more work that needs to be done, or is this ticket about supporting Flink 2.1 so we can leverage it?

Hi, @polyzos. Fluss introduced prefixLookup in v0.6. However, to enable end-to-end deltaJoin capabilities on Flink, the Fluss connector layer needs to integrate with Flink's deltaJoin API, which wasn't introduced until Flink 2.1 (see FLIP-486: Introduce A New DeltaJoin). Therefore, Fluss need to implement this integration to achieve true end-to-end deltaJoin functionality. And @xuyangzhong （The implementer of FLIP-486）will take on this task to help Fluss achieve this capability.

[wuchong]

@xuyangzhong Flink 2.1 has been supported in #1176. Could you help to implement Delta Join for fluss?

[zhanp0304]

Hi @xuyangzhong, @swuferhong

I noticed that the JIRA ticket FLINK-38029 Release Testing: Verify FLIP-486: Introduce a new DeltaJoin
has been updated to 'RESOLVED'. Does this mean that Fluss 0.6/0.7 can now integrate with Flink 2.1 to leverage the Delta Join operator optimization?

Specifically, when we set 'table.optimizer.delta-join.strategy' = 'force' in Flink SQL Job, will the Flink planner automatically optimize the execution plan to use the Delta Join operator as described in the JIRA?

Any insights on the compatibility and configuration details would be greatly appreciated!

[xuyangzhong]

Hi @xuyangzhong, @swuferhong

I noticed that the JIRA ticket FLINK-38029 Release Testing: Verify FLIP-486: Introduce a new DeltaJoin has been updated to 'RESOLVED'. Does this mean that Fluss 0.6/0.7 can now integrate with Flink 2.1 to leverage the Delta Join operator optimization?

Specifically, when we set 'table.optimizer.delta-join.strategy' = 'force' in Flink SQL Job, will the Flink planner automatically optimize the execution plan to use the Delta Join operator as described in the JIRA?

Any insights on the compatibility and configuration details would be greatly appreciated!

Hi, @zhanp0304. Fluss need to implement certain interfaces to support Flink Delta Join, which is the purpose of this issue. So it will work after releasing v0.8.

[xuyangzhong]

Hi, @wuchong @swuferhong, the pr is available and the ci is green. Could you help review it if you have time?