[Feature](Streaming Job) Extend streaming job to support MySQL synchronization#58898

JNSimba · 2025-12-10T07:52:42Z

What problem does this PR solve?

Issue Number: close #58896

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…ql-cdc

# Conflicts: # build.sh

Thearas · 2025-12-10T07:52:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR extends streaming jobs to support MySQL synchronization via CDC (Change Data Capture), enabling users to sync data from MySQL databases to Doris in real-time. The implementation includes a new CDC client service and modifications to the streaming job framework.

Key Changes:

Introduces a CDC client Spring Boot application that interfaces with MySQL using Flink CDC connectors
Adds support for FROM MySQL TO Database syntax in job creation
Implements split-based data reading for both snapshot and binlog phases
Adds RPC endpoints for BE-FE communication to handle CDC operations

Reviewed changes

Copilot reviewed 85 out of 85 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job.groovy	Regression test for MySQL streaming job with CDC
gensrc/proto/internal_service.proto	Adds RPC interface for CDC client communication
fs_brokers/cdc_client/**	Complete CDC client implementation using Spring Boot
fe/fe-core/.../streaming/**	Extends streaming job framework with multi-table task support
fe/fe-core/.../offset/jdbc/**	JDBC offset provider for tracking MySQL binlog positions
fe/fe-core/.../util/StreamingJobUtils.java	Utility functions for streaming job management
docker/thirdparties/docker-compose/mysql/my.cnf	Enables MySQL binlog for CDC

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…59705) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #58898

…pty tables (#59735) ### What problem does this PR solve? Fix the issue of synchronization failure under empty tables Related PR: #58898

### What problem does this PR solve? fix show task error info when task timeout Related PR: #58898

…59760) ### What problem does this PR solve? fix get remote meta failed to pause streaming job Releate PR: #58898

### What problem does this PR solve? fix show task error info when task timeout Related PR: #58898

…59760) ### What problem does this PR solve? fix get remote meta failed to pause streaming job Releate PR: #58898

…pache#59705) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#58898

…pty tables (apache#59735) ### What problem does this PR solve? Fix the issue of synchronization failure under empty tables Related PR: apache#58898

…e#59784) ### What problem does this PR solve? fix show task error info when task timeout Related PR: apache#58898

…pache#59760) ### What problem does this PR solve? fix get remote meta failed to pause streaming job Releate PR: apache#58898

…#59828) ### What problem does this PR solve? Related PR: #58898 1. The length of varchar needs to be multiplied by 3 when creating the table. 2. Columns are ordered according to the primary key. 3. Unsupported column types will result in an error.

… remainsplit relay problem (#59883) ### What problem does this PR solve? Related PR: #58898 After the Job is created for the first time, starting from the initial offset, the task for the first split is scheduled, When the task status is running or failed, If FE restarts, the split needs to be restore from the meta again.

### What problem does this PR solve? Related PR: #58898 #59461 This PR primarily optimizes the speed of incremental and snapshot reads. 1. For incremental reads: - Binding the fetch logic to an interval allows fetching data within that interval. - Splitting the fetch and write logic asynchronously. 2. For snapshot reads: - Introducing the `snapshot_split_size` and `snapshot_parallelism` parameters. - `snapshot_split_size`: Adjusts the size of each chunk during the split phase, allowing each split to fetch more data. - `snapshot_parallelism`: The degree of parallelism during the snapshot read phase, i.e., how many chunks can run simultaneously, and how many chunks are scheduled in a single task.

…l/pg streaming job (#60473) ### What problem does this PR solve? Related PR: #58898 #59461 In some scenarios, it is necessary to tolerate a certain amount of erroneous data. Supported parameters: `load.strict_mode`: Whether to enable strict mode, defaults to false. `load.max_filter_ratio`: The maximum allowed filtering rate within the sampling window, defaults to zero tolerance. The sampling window is `max_interval * 10`. That is, if the number of erroneous rows/total rows exceeds `max_filter_ratio` within the sampling window, the job will be paused, requiring manual intervention to check data quality issues. eg: ``` CREATE JOB test_streaming_mysql_job_errormsg ON STREAMING FROM MYSQL ( "jdbc_url" = "jdbc:mysql://127.0.0.1:3308", ...... ) TO DATABASE database ( "table.create.properties.replication_num" = "1" ... "load.max_filter_ratio" = "1" ) ```

JNSimba and others added 19 commits November 24, 2025 20:26

add cdc client read data

37c8553

add a part of stream load sink

cd4e1e9

fix write split chunk api

4b56052

add be request cdc client

03fa875

add fe rpc interface and test

aa85742

fix cdc client manager compile

c12ca09

Merge branch 'mysql-cdc' of https://github.com/JNSimba/doris into mys…

3f90ae6

…ql-cdc

add streaming job split chunks and streaming multi task

56ac92c

add fe create streamt multi ask

38507ae

fix be forward request

3c2d358

add multi table sync and checkstyle

124d2ae

add build script

06c8ae7

update check style

8f8f676

fix be fork

59a5cf2

fix offset bug

2b783cd

Merge branch 'mysql-cdc' of https://github.com/JNSimba/doris into mys…

6b2443f

…ql-cdc

fix create table and offset consumer bug

79a71ed

add case for mysql sync

4f3a1e5

Merge branch 'master' into mysql-cdc

8d9b14b

# Conflicts: # build.sh

JNSimba requested review from CalvinKirs and morningman as code owners December 10, 2025 07:52

JNSimba changed the title ~~[Proposal] Extend streaming job to support MySQL synchronization~~ [Feature] Extend streaming job to support MySQL synchronization Dec 10, 2025

JNSimba changed the title ~~[Feature] Extend streaming job to support MySQL synchronization~~ [Feature](Streaming Job) Extend streaming job to support MySQL synchronization Dec 10, 2025

code style

f63e984

JNSimba requested review from Copilot and liaoxin01 December 10, 2025 08:47

Copilot started reviewing on behalf of JNSimba December 10, 2025 08:48 View session

Copilot AI reviewed Dec 10, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

JNSimba mentioned this pull request Jan 9, 2026

[Fix](streamingjob) modify select backend policy for streaming job #59705

Merged

16 tasks

JNSimba added a commit that referenced this pull request Jan 9, 2026

[Fix](streamingjob) modify select backend policy for streaming job (#…

85b1956

…59705) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #58898

github-actions bot pushed a commit that referenced this pull request Jan 9, 2026

[Fix](streamingjob) modify select backend policy for streaming job (#…

20e0261

…59705) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #58898

This was referenced Jan 9, 2026

[Fix](Streamingjob) Fix the issue of synchronization failure under empty tables #59735

Merged

[Fix](Streamingjob) fix show task error info when task timeout #59784

Merged

[Fix](Streaming) fix get remote meta failed to pause streaming job #59760

Merged

JNSimba added a commit that referenced this pull request Jan 12, 2026

[Fix](Streamingjob) Fix the issue of synchronization failure under em…

7919f2a

…pty tables (#59735) ### What problem does this PR solve? Fix the issue of synchronization failure under empty tables Related PR: #58898

JNSimba added a commit that referenced this pull request Jan 13, 2026

[Fix](Streamingjob) fix show task error info when task timeout (#59784)

95b7c22

### What problem does this PR solve? fix show task error info when task timeout Related PR: #58898

JNSimba added a commit that referenced this pull request Jan 13, 2026

[Fix](Streaming) fix get remote meta failed to pause streaming job (#…

4bf66fb

…59760) ### What problem does this PR solve? fix get remote meta failed to pause streaming job Releate PR: #58898

github-actions bot pushed a commit that referenced this pull request Jan 13, 2026

[Fix](Streamingjob) fix show task error info when task timeout (#59784)

fdc83a4

### What problem does this PR solve? fix show task error info when task timeout Related PR: #58898

github-actions bot pushed a commit that referenced this pull request Jan 13, 2026

[Fix](Streaming) fix get remote meta failed to pause streaming job (#…

0691e4d

…59760) ### What problem does this PR solve? fix get remote meta failed to pause streaming job Releate PR: #58898

This was referenced Jan 13, 2026

[Feature](Streaming Job) Extend streaming job to support Postgres synchronization #59461

Merged

[Fix](StreamingJob) fix create table issues when create streaming job #59828

Merged

zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Jan 13, 2026

[Fix](streamingjob) modify select backend policy for streaming job (a…

c792d71

…pache#59705) ### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#58898

zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Jan 13, 2026

[Fix](Streamingjob) fix show task error info when task timeout (apach…

cd38591

…e#59784) ### What problem does this PR solve? fix show task error info when task timeout Related PR: apache#58898

JNSimba mentioned this pull request Jan 14, 2026

[Fix](StreamingJob) fix the first split task scheduled and fe restart remainsplit relay problem #59883

Merged

16 tasks

This was referenced Jan 21, 2026

[Feature](tvf) Support cdc stream tvf for mysql and pg #60116

Open

[Fix](StreamingJob) Optimize CDC consumption strategy #60181

Merged

JNSimba mentioned this pull request Feb 3, 2026

[Improve](StreamingJob) add max_filter_ratio and strict mode for mysql/pg streaming job #60473

Merged

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature](Streaming Job) Extend streaming job to support MySQL synchronization#58898