Skip to content

[fix](job) fix transaction timeout do not match routine load task adaptive timeout#57967

Merged
liaoxin01 merged 1 commit intoapache:masterfrom
sollhui:fix_rl_task_txn_timeout
Nov 13, 2025
Merged

[fix](job) fix transaction timeout do not match routine load task adaptive timeout#57967
liaoxin01 merged 1 commit intoapache:masterfrom
sollhui:fix_rl_task_txn_timeout

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Nov 12, 2025

What problem does this PR solve?

Fix transaction timeout do not match routine load task adaptive timeout, introduced by #56930

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Nov 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Nov 12, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 12, 2025

TPC-H: Total hot run time: 34446 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c10ee30075f1205daeb660a1b4d198a5b596a18b, data reload: false

------ Round 1 ----------------------------------
q1	17631	5266	5114	5114
q2	2050	319	223	223
q3	10203	1302	722	722
q4	10241	958	381	381
q5	7563	2354	2384	2354
q6	188	170	139	139
q7	932	760	627	627
q8	9360	1365	1213	1213
q9	6988	5158	5071	5071
q10	6841	2218	1814	1814
q11	503	310	290	290
q12	346	379	234	234
q13	17758	3697	3065	3065
q14	233	237	213	213
q15	576	510	502	502
q16	1039	1011	944	944
q17	575	876	358	358
q18	7463	7297	7050	7050
q19	1188	966	557	557
q20	360	349	261	261
q21	4050	3215	2346	2346
q22	1071	1030	968	968
Total cold run time: 107159 ms
Total hot run time: 34446 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5156	5142	5135	5135
q2	255	338	231	231
q3	2153	2685	2265	2265
q4	1382	1805	1346	1346
q5	4224	4621	4634	4621
q6	222	181	135	135
q7	2062	2015	1867	1867
q8	2651	2578	2655	2578
q9	7375	7227	7288	7227
q10	3113	3260	2870	2870
q11	583	522	510	510
q12	715	820	639	639
q13	3590	3917	3385	3385
q14	304	302	285	285
q15	538	504	544	504
q16	1141	1100	1084	1084
q17	1220	1596	1411	1411
q18	8000	7814	7613	7613
q19	807	889	1009	889
q20	2043	2135	1924	1924
q21	5109	4366	4322	4322
q22	1085	1020	994	994
Total cold run time: 53728 ms
Total hot run time: 51835 ms

@doris-robot
Copy link

doris-robot commented Nov 12, 2025

TPC-DS: Total hot run time: 187589 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c10ee30075f1205daeb660a1b4d198a5b596a18b, data reload: false

query1	1029	409	403	403
query2	6569	1709	1708	1708
query3	6767	234	225	225
query4	26809	23336	23388	23336
query5	4363	626	471	471
query6	352	236	219	219
query7	4651	503	295	295
query8	307	259	258	258
query9	8748	2614	2635	2614
query10	486	345	297	297
query11	15174	15121	14740	14740
query12	182	128	114	114
query13	1693	580	445	445
query14	10396	9283	9135	9135
query15	202	188	178	178
query16	7665	655	521	521
query17	1242	757	613	613
query18	2012	410	319	319
query19	221	202	171	171
query20	129	123	121	121
query21	215	131	112	112
query22	4146	4058	3929	3929
query23	33906	32891	32860	32860
query24	8459	2349	2438	2349
query25	598	517	434	434
query26	1241	276	162	162
query27	2754	491	347	347
query28	4391	2222	2204	2204
query29	812	607	485	485
query30	290	229	199	199
query31	923	824	689	689
query32	79	79	75	75
query33	606	377	332	332
query34	810	828	519	519
query35	801	831	735	735
query36	974	1006	893	893
query37	118	116	88	88
query38	3577	3479	3542	3479
query39	1523	1404	1418	1404
query40	224	131	117	117
query41	62	63	63	63
query42	128	115	108	108
query43	501	495	480	480
query44	1260	747	728	728
query45	188	181	174	174
query46	887	989	641	641
query47	1740	1810	1711	1711
query48	411	433	312	312
query49	798	526	429	429
query50	674	696	403	403
query51	3845	3863	3956	3863
query52	115	113	109	109
query53	250	275	194	194
query54	315	293	275	275
query55	90	89	82	82
query56	322	332	304	304
query57	1192	1182	1139	1139
query58	290	277	269	269
query59	2578	2634	2541	2541
query60	362	361	333	333
query61	195	197	191	191
query62	805	730	678	678
query63	236	203	194	194
query64	4683	1274	981	981
query65	4038	3963	3946	3946
query66	1188	465	357	357
query67	15317	14981	14860	14860
query68	8180	950	605	605
query69	516	340	291	291
query70	1368	1268	1206	1206
query71	446	336	312	312
query72	6149	4964	4936	4936
query73	634	592	355	355
query74	9065	9017	9019	9017
query75	3482	3303	2839	2839
query76	3442	1167	778	778
query77	570	427	328	328
query78	9462	9735	8856	8856
query79	2038	843	597	597
query80	681	570	499	499
query81	496	276	241	241
query82	425	164	134	134
query83	274	272	257	257
query84	272	108	96	96
query85	919	476	441	441
query86	342	303	289	289
query87	3664	3718	3612	3612
query88	3501	2253	2233	2233
query89	399	334	303	303
query90	1923	226	226	226
query91	170	166	136	136
query92	84	71	61	61
query93	1243	965	651	651
query94	759	452	348	348
query95	414	324	317	317
query96	490	580	281	281
query97	2909	2982	2846	2846
query98	246	212	212	212
query99	1420	1419	1326	1326
Total cold run time: 273897 ms
Total hot run time: 187589 ms

@doris-robot
Copy link

doris-robot commented Nov 12, 2025

ClickBench: Total hot run time: 27.29 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c10ee30075f1205daeb660a1b4d198a5b596a18b, data reload: false

query1	0.05	0.06	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.27	0.26	0.25
query6	1.15	0.64	0.64
query7	0.03	0.03	0.02
query8	0.05	0.05	0.04
query9	0.60	0.54	0.52
query10	0.58	0.58	0.58
query11	0.17	0.12	0.11
query12	0.14	0.12	0.12
query13	0.62	0.60	0.60
query14	1.00	1.01	1.00
query15	0.85	0.84	0.83
query16	0.40	0.39	0.39
query17	1.05	1.02	1.06
query18	0.21	0.19	0.20
query19	1.88	1.81	1.84
query20	0.01	0.02	0.01
query21	15.43	0.20	0.13
query22	5.06	0.07	0.04
query23	15.66	0.26	0.10
query24	3.00	0.53	0.27
query25	0.08	0.06	0.06
query26	0.14	0.14	0.12
query27	0.07	0.05	0.06
query28	4.62	1.13	0.93
query29	12.60	4.03	3.25
query30	0.28	0.14	0.12
query31	2.81	0.57	0.38
query32	3.23	0.55	0.47
query33	3.12	3.03	3.04
query34	15.73	5.20	4.54
query35	4.60	4.53	4.59
query36	0.67	0.50	0.49
query37	0.10	0.07	0.07
query38	0.07	0.04	0.03
query39	0.04	0.04	0.03
query40	0.18	0.13	0.14
query41	0.08	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.65 s
Total hot run time: 27.29 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 12, 2025

FE Regression Coverage Report

Increment line coverage 87.50% (7/8) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Nov 12, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 12, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 12, 2025

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes Nov 13, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 4ce3e6f into apache:master Nov 13, 2025
29 of 30 checks passed
sollhui added a commit to sollhui/doris that referenced this pull request Dec 9, 2025
@sollhui
[fix](job) fix transaction timeout do not match routine load task ada…
77fa7f2 
…ptive timeout (apache#57967)

Fix transaction timeout do not match routine load task adaptive timeout,
introduced by apache#56930
@sollhui sollhui mentioned this pull request Dec 9, 2025
Merged
16 tasks
yiguolei pushed a commit that referenced this pull request Dec 17, 2025
@sollhui
branch-4.0: [enhance](job) support adaptive batch param for routine l…
471f747 
…oad job (#58846)

### What problem does this PR solve?

pick #56930 and
#57967

Users may set the max batch interval relatively small for visibility,
which may result in insufficient throughput and data backlog when
traffic is high. We propose an adaptive max batch interval scheme aimed
at prioritizing throughput over visibility during data backlog

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@sollhui @Thearas @doris-robot @hello-stephen @liaoxin01 @dataroaring @yiguolei

Comments