Skip to content

[enhance](job) support adaptive batch param for routine load job#56930

Merged
liaoxin01 merged 2 commits intoapache:masterfrom
sollhui:adaptive_max_batch_interval
Oct 20, 2025
Merged

[enhance](job) support adaptive batch param for routine load job#56930
liaoxin01 merged 2 commits intoapache:masterfrom
sollhui:adaptive_max_batch_interval

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Oct 14, 2025

What problem does this PR solve?

Users may set the max batch interval relatively small for visibility, which may result in insufficient throughput and data backlog when traffic is high. We propose an adaptive max batch interval scheme aimed at prioritizing throughput over visibility during data backlog

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from ad32fe5 to 9fa7d4d Compare October 14, 2025 03:40
@sollhui
Copy link
Contributor Author

sollhui commented Oct 14, 2025

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 190198 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9fa7d4dd3f613728922cf9e99bc358897f236cce, data reload: false

query1	1076	445	435	435
query2	6564	1690	1695	1690
query3	6751	220	226	220
query4	25924	23459	22976	22976
query5	4495	648	494	494
query6	331	263	213	213
query7	4654	503	285	285
query8	295	257	240	240
query9	8697	2593	2570	2570
query10	505	330	294	294
query11	15345	15010	14833	14833
query12	191	121	123	121
query13	1691	563	438	438
query14	11508	9346	9304	9304
query15	208	196	181	181
query16	7681	698	487	487
query17	1622	785	619	619
query18	2027	415	316	316
query19	213	268	175	175
query20	141	141	150	141
query21	230	138	123	123
query22	4772	4771	4676	4676
query23	34972	34119	33932	33932
query24	8558	2670	2469	2469
query25	567	547	462	462
query26	1240	290	179	179
query27	2732	517	393	393
query28	4763	2203	2183	2183
query29	790	670	568	568
query30	336	251	230	230
query31	938	895	756	756
query32	84	86	83	83
query33	614	431	362	362
query34	863	900	526	526
query35	879	890	826	826
query36	1043	1028	879	879
query37	119	107	85	85
query38	3542	3551	3513	3513
query39	1480	1453	1411	1411
query40	230	125	116	116
query41	61	58	61	58
query42	130	111	110	110
query43	477	486	493	486
query44	1321	831	820	820
query45	218	177	171	171
query46	831	996	630	630
query47	1773	1829	1726	1726
query48	381	414	319	319
query49	751	481	415	415
query50	637	690	413	413
query51	3883	3903	3946	3903
query52	105	111	101	101
query53	234	281	196	196
query54	602	596	527	527
query55	93	86	84	84
query56	340	328	311	311
query57	1203	1193	1136	1136
query58	282	274	275	274
query59	2576	2653	2482	2482
query60	351	342	325	325
query61	156	156	149	149
query62	800	738	692	692
query63	231	194	189	189
query64	4418	1154	842	842
query65	4065	3970	3992	3970
query66	1106	469	351	351
query67	15700	15319	15181	15181
query68	9457	931	592	592
query69	490	322	286	286
query70	1393	1339	1248	1248
query71	489	327	315	315
query72	5553	4822	4761	4761
query73	687	549	361	361
query74	9117	9103	8956	8956
query75	4506	3378	2867	2867
query76	3797	1168	741	741
query77	1002	404	319	319
query78	9525	9791	8932	8932
query79	3333	842	575	575
query80	722	576	505	505
query81	487	258	227	227
query82	428	165	136	136
query83	294	272	253	253
query84	301	118	91	91
query85	882	476	505	476
query86	373	314	293	293
query87	3794	3810	3665	3665
query88	2778	2238	2238	2238
query89	437	324	295	295
query90	2127	216	289	216
query91	168	166	135	135
query92	81	75	65	65
query93	2204	966	641	641
query94	682	436	328	328
query95	407	325	318	318
query96	478	591	279	279
query97	2915	3031	2886	2886
query98	233	216	211	211
query99	1438	1415	1322	1322
Total cold run time: 281974 ms
Total hot run time: 190198 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9fa7d4dd3f613728922cf9e99bc358897f236cce, data reload: false

query1	0.06	0.04	0.05
query2	0.09	0.05	0.06
query3	0.25	0.09	0.08
query4	1.60	0.12	0.12
query5	0.28	0.26	0.24
query6	1.17	0.65	0.66
query7	0.04	0.03	0.03
query8	0.05	0.04	0.05
query9	0.63	0.53	0.51
query10	0.58	0.60	0.58
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.63	0.61	0.62
query14	1.04	1.02	1.02
query15	0.87	0.86	0.87
query16	0.41	0.39	0.39
query17	1.06	1.04	1.04
query18	0.22	0.20	0.20
query19	1.89	1.91	1.82
query20	0.02	0.01	0.01
query21	15.43	0.92	0.58
query22	0.77	1.23	0.63
query23	14.92	1.33	0.61
query24	6.71	1.83	1.06
query25	0.47	0.19	0.15
query26	0.64	0.16	0.13
query27	0.06	0.06	0.05
query28	9.26	1.35	0.92
query29	12.56	3.90	3.23
query30	0.28	0.14	0.13
query31	2.83	0.61	0.38
query32	3.26	0.57	0.47
query33	3.10	3.10	3.17
query34	16.28	5.52	4.81
query35	4.91	4.92	4.90
query36	0.69	0.52	0.51
query37	0.11	0.07	0.08
query38	0.07	0.05	0.05
query39	0.04	0.03	0.03
query40	0.19	0.15	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.02
query43	0.05	0.04	0.04
Total cold run time: 103.96 s
Total hot run time: 30.55 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/14) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 78.57% (11/14) 🎉
Increment coverage report
Complete coverage report

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from 9fa7d4d to 56a93ef Compare October 14, 2025 07:49
@sollhui
Copy link
Contributor Author

sollhui commented Oct 14, 2025

run buildall

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from 56a93ef to ea72d72 Compare October 14, 2025 08:07
@sollhui
Copy link
Contributor Author

sollhui commented Oct 14, 2025

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 190175 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea72d72fcced291d875c6d33ffc85535f04073c2, data reload: false

query1	1069	433	427	427
query2	6556	1690	1664	1664
query3	6774	222	218	218
query4	26712	23335	23474	23335
query5	5096	616	474	474
query6	322	237	211	211
query7	4641	503	297	297
query8	317	268	255	255
query9	8740	2576	2556	2556
query10	518	344	277	277
query11	15572	14980	14855	14855
query12	186	117	115	115
query13	1663	561	423	423
query14	11574	9192	9089	9089
query15	224	200	174	174
query16	7660	661	513	513
query17	1247	756	614	614
query18	2028	419	393	393
query19	248	235	201	201
query20	147	139	138	138
query21	217	136	112	112
query22	4916	4683	4656	4656
query23	34757	34019	33683	33683
query24	8587	2463	2523	2463
query25	622	561	512	512
query26	1333	291	167	167
query27	3184	509	373	373
query28	4741	2238	2302	2238
query29	842	656	546	546
query30	310	236	207	207
query31	974	839	784	784
query32	89	79	71	71
query33	600	379	341	341
query34	834	873	537	537
query35	832	896	792	792
query36	1019	1043	947	947
query37	127	118	88	88
query38	3520	3496	3467	3467
query39	1481	1402	1382	1382
query40	252	125	117	117
query41	62	58	58	58
query42	123	110	112	110
query43	494	499	472	472
query44	1305	826	803	803
query45	187	175	172	172
query46	839	990	624	624
query47	1792	1829	1768	1768
query48	406	441	312	312
query49	776	491	395	395
query50	645	674	405	405
query51	3908	3859	3925	3859
query52	117	111	99	99
query53	236	264	189	189
query54	600	591	521	521
query55	89	86	86	86
query56	310	329	309	309
query57	1195	1197	1134	1134
query58	290	283	289	283
query59	2557	2649	2584	2584
query60	352	352	326	326
query61	168	163	161	161
query62	816	731	694	694
query63	239	190	191	190
query64	4536	1254	943	943
query65	4046	4017	4174	4017
query66	1067	435	359	359
query67	15569	15280	15088	15088
query68	9520	877	592	592
query69	485	319	284	284
query70	1421	1267	1251	1251
query71	501	339	326	326
query72	5807	4879	4833	4833
query73	714	575	353	353
query74	8875	9103	8855	8855
query75	4573	3376	2832	2832
query76	4330	1166	749	749
query77	1014	401	330	330
query78	9638	9810	8941	8941
query79	1661	880	581	581
query80	705	569	493	493
query81	494	262	225	225
query82	457	160	134	134
query83	293	279	256	256
query84	307	119	91	91
query85	886	475	413	413
query86	354	330	300	300
query87	3912	3688	3636	3636
query88	2857	2271	2285	2271
query89	408	334	311	311
query90	2015	220	233	220
query91	156	167	137	137
query92	83	69	71	69
query93	1142	989	647	647
query94	685	454	318	318
query95	411	319	320	319
query96	480	588	281	281
query97	2964	2982	2891	2891
query98	234	209	212	209
query99	1464	1410	1354	1354
Total cold run time: 282101 ms
Total hot run time: 190175 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ea72d72fcced291d875c6d33ffc85535f04073c2, data reload: false

query1	0.06	0.04	0.04
query2	0.09	0.05	0.06
query3	0.24	0.08	0.08
query4	1.60	0.11	0.12
query5	0.27	0.27	0.25
query6	1.17	0.66	0.64
query7	0.04	0.03	0.03
query8	0.06	0.05	0.04
query9	0.63	0.53	0.53
query10	0.58	0.58	0.57
query11	0.19	0.11	0.11
query12	0.15	0.12	0.12
query13	0.64	0.61	0.62
query14	1.04	1.05	1.00
query15	0.87	0.86	0.89
query16	0.44	0.39	0.40
query17	1.09	1.08	1.06
query18	0.21	0.19	0.21
query19	2.00	1.87	1.83
query20	0.02	0.01	0.02
query21	15.42	0.93	0.57
query22	0.76	1.20	0.66
query23	14.93	1.38	0.66
query24	7.12	1.38	0.57
query25	0.56	0.30	0.10
query26	0.57	0.16	0.13
query27	0.06	0.06	0.05
query28	9.75	1.36	0.95
query29	12.55	3.95	3.31
query30	0.28	0.14	0.11
query31	2.84	0.59	0.38
query32	3.25	0.56	0.47
query33	3.10	3.08	3.10
query34	16.21	5.49	4.83
query35	4.96	4.95	4.94
query36	0.69	0.52	0.50
query37	0.10	0.08	0.08
query38	0.06	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 104.98 s
Total hot run time: 30.21 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 87.50% (14/16) 🎉
Increment coverage report
Complete coverage report

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from ea72d72 to 7ad9ede Compare October 16, 2025 13:27
@sollhui
Copy link
Contributor Author

sollhui commented Oct 16, 2025

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 190518 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ad9ede105c99a2531b84c6c94ba10d230e46e8f, data reload: false

query1	1070	445	410	410
query2	6567	1720	1664	1664
query3	6747	221	222	221
query4	26590	23726	23564	23564
query5	5277	634	497	497
query6	346	263	237	237
query7	4672	497	309	309
query8	316	271	269	269
query9	8715	2565	2580	2565
query10	522	366	295	295
query11	15581	15167	14758	14758
query12	201	122	115	115
query13	1691	551	448	448
query14	11618	9244	9287	9244
query15	272	204	192	192
query16	7959	739	544	544
query17	1710	930	727	727
query18	2251	458	365	365
query19	313	216	190	190
query20	143	142	130	130
query21	221	146	128	128
query22	5046	5206	5027	5027
query23	34519	34016	33725	33725
query24	8567	2509	2528	2509
query25	578	510	463	463
query26	1285	276	158	158
query27	2740	564	356	356
query28	4823	2221	2217	2217
query29	951	738	505	505
query30	313	236	205	205
query31	930	855	812	812
query32	84	71	71	71
query33	571	389	378	378
query34	829	910	537	537
query35	824	853	745	745
query36	944	986	870	870
query37	128	112	87	87
query38	3485	3539	3502	3502
query39	1473	1438	1387	1387
query40	219	124	114	114
query41	60	56	59	56
query42	121	106	110	106
query43	495	489	479	479
query44	1336	827	819	819
query45	183	176	174	174
query46	836	984	636	636
query47	1746	1830	1729	1729
query48	379	406	328	328
query49	762	491	400	400
query50	636	682	409	409
query51	4062	3949	3893	3893
query52	109	110	97	97
query53	237	265	194	194
query54	617	591	520	520
query55	97	84	82	82
query56	323	318	310	310
query57	1202	1182	1142	1142
query58	295	281	281	281
query59	2523	2588	2578	2578
query60	346	341	322	322
query61	155	150	145	145
query62	819	723	660	660
query63	239	189	188	188
query64	4467	1206	820	820
query65	4042	3959	3960	3959
query66	1081	463	326	326
query67	15174	15038	14868	14868
query68	9376	944	604	604
query69	496	327	287	287
query70	1399	1354	1324	1324
query71	492	345	326	326
query72	5465	4813	4760	4760
query73	669	551	361	361
query74	8894	9019	9022	9019
query75	4421	3367	2861	2861
query76	4674	1232	721	721
query77	1022	417	305	305
query78	9588	9912	8876	8876
query79	2054	812	586	586
query80	712	549	521	521
query81	496	264	225	225
query82	476	158	135	135
query83	285	262	249	249
query84	350	113	91	91
query85	897	479	425	425
query86	347	325	294	294
query87	3764	3743	3594	3594
query88	2833	2236	2212	2212
query89	407	326	300	300
query90	2032	226	234	226
query91	163	167	132	132
query92	86	73	70	70
query93	1693	963	643	643
query94	690	440	327	327
query95	407	321	317	317
query96	480	582	287	287
query97	2969	2964	2885	2885
query98	240	215	228	215
query99	1468	1448	1279	1279
Total cold run time: 282778 ms
Total hot run time: 190518 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7ad9ede105c99a2531b84c6c94ba10d230e46e8f, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.06	0.06
query3	0.25	0.08	0.09
query4	1.61	0.12	0.12
query5	0.28	0.26	0.26
query6	1.16	0.65	0.64
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.63	0.53	0.53
query10	0.58	0.58	0.58
query11	0.17	0.12	0.11
query12	0.18	0.12	0.12
query13	0.62	0.63	0.61
query14	1.03	1.02	1.03
query15	0.92	0.84	0.85
query16	0.42	0.40	0.40
query17	1.03	1.04	1.06
query18	0.21	0.21	0.20
query19	1.92	1.86	1.82
query20	0.01	0.02	0.01
query21	15.45	0.95	0.57
query22	0.76	1.16	0.64
query23	15.03	1.39	0.63
query24	7.50	0.69	1.10
query25	0.46	0.25	0.09
query26	0.54	0.16	0.14
query27	0.07	0.05	0.05
query28	9.39	1.33	0.92
query29	12.60	4.02	3.31
query30	0.29	0.14	0.11
query31	2.83	0.60	0.39
query32	3.26	0.57	0.49
query33	3.06	3.13	3.06
query34	16.19	5.47	4.84
query35	4.91	4.91	4.90
query36	0.70	0.52	0.53
query37	0.11	0.08	0.08
query38	0.07	0.05	0.05
query39	0.04	0.03	0.04
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.84 s
Total hot run time: 30.3 s

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from 7ad9ede to 2283a48 Compare October 16, 2025 15:39
@sollhui
Copy link
Contributor Author

sollhui commented Oct 16, 2025

run buildall

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from 2283a48 to 43ef5e0 Compare October 16, 2025 15:43
@sollhui
Copy link
Contributor Author

sollhui commented Oct 16, 2025

run buildall

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from 43ef5e0 to 4aeafb4 Compare October 16, 2025 15:51
@sollhui
Copy link
Contributor Author

sollhui commented Oct 16, 2025

run buildall

@sollhui sollhui force-pushed the adaptive_max_batch_interval branch from 4aeafb4 to dce28d8 Compare October 16, 2025 16:10
@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/15) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (15/15) 🎉
Increment coverage report
Complete coverage report

3 similar comments
@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (15/15) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (15/15) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (15/15) 🎉
Increment coverage report
Complete coverage report

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2025
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit f34b8c1 into apache:master Oct 20, 2025
29 of 30 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 20, 2025
)

### What problem does this PR solve?

Users may set the `max batch interval` relatively small for visibility,
which may result in insufficient throughput and data backlog when
traffic is high. We propose an adaptive max batch interval scheme aimed
at **prioritizing throughput over visibility during data backlog**
github-actions bot pushed a commit that referenced this pull request Oct 20, 2025
)

### What problem does this PR solve?

Users may set the `max batch interval` relatively small for visibility,
which may result in insufficient throughput and data backlog when
traffic is high. We propose an adaptive max batch interval scheme aimed
at **prioritizing throughput over visibility during data backlog**
yiguolei pushed a commit that referenced this pull request Oct 22, 2025
)

### What problem does this PR solve?

Users may set the `max batch interval` relatively small for visibility,
which may result in insufficient throughput and data backlog when
traffic is high. We propose an adaptive max batch interval scheme aimed
at **prioritizing throughput over visibility during data backlog**
liaoxin01 pushed a commit that referenced this pull request Nov 13, 2025
…ptive timeout (#57967)

Fix transaction timeout do not match routine load task adaptive timeout,
introduced by #56930
sollhui added a commit to sollhui/doris that referenced this pull request Dec 9, 2025
…che#56930)

### What problem does this PR solve?

Users may set the `max batch interval` relatively small for visibility,
which may result in insufficient throughput and data backlog when
traffic is high. We propose an adaptive max batch interval scheme aimed
at **prioritizing throughput over visibility during data backlog**
sollhui added a commit to sollhui/doris that referenced this pull request Dec 9, 2025
…ptive timeout (apache#57967)

Fix transaction timeout do not match routine load task adaptive timeout,
introduced by apache#56930
yiguolei pushed a commit that referenced this pull request Dec 17, 2025
…oad job (#58846)

### What problem does this PR solve?

pick #56930 and
#57967

Users may set the max batch interval relatively small for visibility,
which may result in insufficient throughput and data backlog when
traffic is high. We propose an adaptive max batch interval scheme aimed
at prioritizing throughput over visibility during data backlog

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
JNSimba pushed a commit that referenced this pull request Jan 20, 2026
### What problem does this PR solve?

## Problem

When routine load uses adaptive batch interval (introduced in #56930),
the FE transaction timeout does not match the BE task timeout, causing
transactions to be aborted by `txnCleaner` while BE is still processing.

### Root Cause

In `RoutineLoadTaskScheduler.scheduleOneTask()`, the execution order is:
1. `beginTxn()` - creates transaction with **original** `timeoutMs`
2. `createRoutineLoadTask()` → `adaptiveBatchParam()` - updates
`timeoutMs` to **adaptive** value

This causes a mismatch:
- FE transaction timeout: original value (e.g., 200 seconds)
- BE task timeout: adaptive value (e.g., 360 seconds)
github-actions bot pushed a commit that referenced this pull request Jan 20, 2026
### What problem does this PR solve?

## Problem

When routine load uses adaptive batch interval (introduced in #56930),
the FE transaction timeout does not match the BE task timeout, causing
transactions to be aborted by `txnCleaner` while BE is still processing.

### Root Cause

In `RoutineLoadTaskScheduler.scheduleOneTask()`, the execution order is:
1. `beginTxn()` - creates transaction with **original** `timeoutMs`
2. `createRoutineLoadTask()` → `adaptiveBatchParam()` - updates
`timeoutMs` to **adaptive** value

This causes a mismatch:
- FE transaction timeout: original value (e.g., 200 seconds)
- BE task timeout: adaptive value (e.g., 360 seconds)
dataroaring pushed a commit that referenced this pull request Feb 11, 2026
…ut_value (#60664)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #56930
Problem Summary:

Under the condition where `isEof = false`, the calculation of the
timeout in `updateAdaptiveTimeout `also needs to take into account
`Config.routine_load_task_min_timeout_sec`.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
github-actions bot pushed a commit that referenced this pull request Feb 11, 2026
…ut_value (#60664)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #56930
Problem Summary:

Under the condition where `isEof = false`, the calculation of the
timeout in `updateAdaptiveTimeout `also needs to take into account
`Config.routine_load_task_min_timeout_sec`.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants