Skip to content

branch-3.0: [fix](load) fix missing error url return for stream load #54115#54266

Merged
dataroaring merged 1 commit intobranch-3.0from
auto-pick-54115-branch-3.0
Aug 12, 2025
Merged

branch-3.0: [fix](load) fix missing error url return for stream load #54115#54266
dataroaring merged 1 commit intobranch-3.0from
auto-pick-54115-branch-3.0

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 4, 2025

Cherry-picked from #54115

@liaoxin01
[fix](load) fix missing error url return for stream load (#54115)
3d3197a 
when pipe is cancelled, stream load execution may return early without
error url.
(test_stream_load_with_filtered_rows.groovy:64) - Stream load result: {
    "TxnId": 11,
    "Label": "2bbde37b-0589-4cce-8497-58e25af46590",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
"Message": "[CANCELLED]cancelled: [DATA_QUALITY_ERROR]Encountered
unqualified data, stop processing. Please check if the source data
matches the schema, and consider disabling strict mode or increasing
max_filter_ratio.. cur path: ",
    "NumberTotalRows": 32512,
    "NumberLoadedRows": 32460,
    "NumberFilteredRows": 52,
    "NumberUnselectedRows": 0,
    "LoadBytes": 29494781,
    "LoadTimeMs": 2138,
    "BeginTxnTimeMs": 1,
    "StreamLoadPutTimeMs": 7,
    "ReadDataTimeMs": 24,
    "WriteDataTimeMs": 0,
    "ReceiveDataTimeMs": 1836,
    "CommitAndPublishTimeMs": 0
}
@github-actions github-actions bot requested a review from dataroaring as a code owner August 4, 2025 03:38
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Aug 4, 2025
@dataroaring dataroaring reopened this Aug 4, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-H: Total hot run time: 39747 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3d3197a14a8643a692bddf78cf7d970f0db7c10f, data reload: false

------ Round 1 ----------------------------------
q1	17599	7258	6678	6678
q2	2067	164	174	164
q3	10676	1105	1181	1105
q4	10484	718	756	718
q5	7705	2857	2797	2797
q6	222	138	132	132
q7	983	604	593	593
q8	9408	1934	2012	1934
q9	6648	6428	6415	6415
q10	7009	2220	2316	2220
q11	460	263	263	263
q12	406	202	212	202
q13	17801	2993	2982	2982
q14	235	208	202	202
q15	517	475	465	465
q16	486	381	372	372
q17	978	568	554	554
q18	7309	6745	6698	6698
q19	1398	1066	1017	1017
q20	469	194	197	194
q21	3819	3115	3070	3070
q22	1115	976	972	972
Total cold run time: 107794 ms
Total hot run time: 39747 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6720	6669	6936	6669
q2	326	229	244	229
q3	2930	2950	2953	2950
q4	2029	1788	1787	1787
q5	5688	5765	5709	5709
q6	202	125	128	125
q7	2259	1789	1854	1789
q8	3363	3588	3523	3523
q9	8801	8968	9057	8968
q10	3604	3565	3671	3565
q11	604	508	504	504
q12	876	644	607	607
q13	7419	3335	3296	3296
q14	327	292	276	276
q15	519	474	468	468
q16	508	454	456	454
q17	1888	1661	1623	1623
q18	8417	7744	7716	7716
q19	1669	1524	1487	1487
q20	2095	1872	1859	1859
q21	5212	5034	4957	4957
q22	1166	1061	1049	1049
Total cold run time: 66622 ms
Total hot run time: 59610 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 4, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.90% (11137/26582)
Line Coverage 32.43% (95417/294249)
Region Coverage 31.57% (49284/156127)
Branch Coverage 28.01% (25264/90198)

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

TPC-DS: Total hot run time: 196172 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3d3197a14a8643a692bddf78cf7d970f0db7c10f, data reload: false

query1	1304	945	907	907
query2	6332	1863	1837	1837
query3	10962	4406	4568	4406
query4	33000	23544	23990	23544
query5	4817	458	450	450
query6	294	194	178	178
query7	3995	309	312	309
query8	288	223	212	212
query9	9396	2566	2568	2566
query10	498	247	248	247
query11	18121	15148	15269	15148
query12	159	104	100	100
query13	1547	428	413	413
query14	8764	6644	7469	6644
query15	256	177	193	177
query16	7836	499	512	499
query17	1599	604	579	579
query18	2056	324	323	323
query19	231	159	163	159
query20	127	117	115	115
query21	201	109	105	105
query22	4833	4509	4610	4509
query23	35276	34418	34174	34174
query24	11140	2890	2905	2890
query25	640	426	411	411
query26	1210	173	167	167
query27	3006	356	355	355
query28	7554	2138	2140	2138
query29	804	474	478	474
query30	261	151	153	151
query31	1044	829	826	826
query32	93	56	55	55
query33	772	286	325	286
query34	994	493	502	493
query35	872	753	732	732
query36	1082	960	957	957
query37	123	64	71	64
query38	4033	3965	3978	3965
query39	1495	1485	1466	1466
query40	251	99	97	97
query41	49	48	56	48
query42	109	100	96	96
query43	502	471	458	458
query44	1313	786	795	786
query45	184	171	165	165
query46	1193	746	726	726
query47	2038	1906	1923	1906
query48	468	388	396	388
query49	940	380	389	380
query50	833	413	409	409
query51	7454	7245	7370	7245
query52	100	87	93	87
query53	263	186	181	181
query54	1141	456	472	456
query55	83	73	82	73
query56	264	249	243	243
query57	1341	1210	1193	1193
query58	221	209	209	209
query59	3178	2945	3037	2945
query60	277	260	267	260
query61	114	107	108	107
query62	905	710	684	684
query63	219	185	185	185
query64	4117	684	663	663
query65	3369	3332	3274	3274
query66	880	294	291	291
query67	16541	15597	15387	15387
query68	4589	564	562	562
query69	436	263	254	254
query70	1147	1136	1113	1113
query71	403	255	271	255
query72	6185	4073	3966	3966
query73	761	353	340	340
query74	10804	8949	9229	8949
query75	3376	2630	2657	2630
query76	2618	1088	1059	1059
query77	393	258	272	258
query78	10551	9499	9493	9493
query79	1625	598	588	588
query80	1069	422	414	414
query81	553	217	223	217
query82	913	85	84	84
query83	237	143	141	141
query84	248	80	82	80
query85	1322	291	299	291
query86	388	292	295	292
query87	4451	4239	4313	4239
query88	3543	2350	2321	2321
query89	428	299	286	286
query90	1939	186	181	181
query91	189	147	146	146
query92	57	48	51	48
query93	1903	543	548	543
query94	868	284	298	284
query95	359	250	248	248
query96	612	278	275	275
query97	3326	3145	3196	3145
query98	212	207	205	205
query99	1531	1292	1307	1292
Total cold run time: 302398 ms
Total hot run time: 196172 ms

@doris-robot
Copy link

doris-robot commented Aug 4, 2025

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3d3197a14a8643a692bddf78cf7d970f0db7c10f, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.53	0.53	0.51
query6	1.14	0.74	0.73
query7	0.04	0.02	0.01
query8	0.03	0.04	0.03
query9	0.56	0.50	0.50
query10	0.55	0.57	0.58
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.59	0.61
query14	0.76	0.79	0.78
query15	0.84	0.81	0.82
query16	0.37	0.37	0.38
query17	1.01	1.06	1.03
query18	0.24	0.21	0.21
query19	1.87	1.86	1.87
query20	0.02	0.02	0.02
query21	15.41	0.58	0.58
query22	2.61	2.42	1.74
query23	17.04	0.89	0.83
query24	3.36	2.04	1.31
query25	0.31	0.13	0.19
query26	0.44	0.14	0.14
query27	0.04	0.04	0.05
query28	8.85	0.51	0.46
query29	12.56	3.29	3.26
query30	0.25	0.06	0.05
query31	2.84	0.38	0.39
query32	3.23	0.46	0.46
query33	2.98	2.98	3.02
query34	17.01	4.52	4.54
query35	4.59	4.54	4.58
query36	0.67	0.48	0.47
query37	0.09	0.06	0.07
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.48 s
Total hot run time: 30.64 s

dataroaring
dataroaring approved these changes Aug 12, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit eaf9d26 into branch-3.0 Aug 12, 2025
23 of 25 checks passed
@github-actions github-actions bot deleted the auto-pick-54115-branch-3.0 branch August 12, 2025 02:47
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @doris-robot @dataroaring @liaoxin01