Skip to content

[opt](nereids) optimize parallel of insert command when olap table have auto partitions#54983

Merged
924060929 merged 2 commits intoapache:masterfrom
924060929:opt-insert-auto-partition
Aug 20, 2025
Merged

[opt](nereids) optimize parallel of insert command when olap table have auto partitions#54983
924060929 merged 2 commits intoapache:masterfrom
924060929:opt-insert-auto-partition

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Aug 19, 2025 •
edited
Loading

What problem does this PR solve?

optimize parallel of insert command when olap table have auto partitions

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Aug 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

924060929 commented Aug 19, 2025

run buildall

@924060929 924060929 added dev/2.1.x dev/3.0.x dev/3.1.x usercase Important user case type label labels Aug 19, 2025
@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-H: Total hot run time: 33612 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46054238b355e495bfe468edca43b9db1118157c, data reload: false

------ Round 1 ----------------------------------
q1	17605	5242	5120	5120
q2	1906	283	181	181
q3	10318	1248	703	703
q4	10229	994	534	534
q5	7511	2408	2275	2275
q6	180	159	134	134
q7	882	733	612	612
q8	9300	1337	1009	1009
q9	6904	5109	5018	5018
q10	6928	2385	1982	1982
q11	493	282	275	275
q12	346	343	218	218
q13	17782	3593	2989	2989
q14	234	244	221	221
q15	563	498	480	480
q16	419	415	373	373
q17	592	833	350	350
q18	7345	7082	7075	7075
q19	1092	930	563	563
q20	335	330	215	215
q21	3799	2537	2329	2329
q22	1068	1004	956	956
Total cold run time: 105831 ms
Total hot run time: 33612 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5206	5128	5157	5128
q2	240	318	220	220
q3	2193	2664	2278	2278
q4	1321	1756	1306	1306
q5	4201	4302	4437	4302
q6	214	182	140	140
q7	2061	1925	1824	1824
q8	2655	2527	2478	2478
q9	7286	7333	7279	7279
q10	3129	3285	2889	2889
q11	593	518	531	518
q12	691	779	606	606
q13	3482	3936	3275	3275
q14	297	306	289	289
q15	505	492	478	478
q16	461	493	450	450
q17	1220	1559	1430	1430
q18	7787	7744	7562	7562
q19	851	919	980	919
q20	1951	2045	1829	1829
q21	5098	4356	4346	4346
q22	1095	1012	991	991
Total cold run time: 52537 ms
Total hot run time: 50537 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-DS: Total hot run time: 184423 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46054238b355e495bfe468edca43b9db1118157c, data reload: false

query1	985	381	427	381
query2	6532	1715	1734	1715
query3	6736	217	224	217
query4	26055	23681	22886	22886
query5	4371	607	512	512
query6	307	210	210	210
query7	4629	495	290	290
query8	285	241	227	227
query9	8580	2842	2871	2842
query10	478	331	288	288
query11	15394	15221	14766	14766
query12	160	116	117	116
query13	1671	558	427	427
query14	8820	5705	5734	5705
query15	203	202	176	176
query16	7168	665	495	495
query17	1213	723	597	597
query18	1996	436	334	334
query19	195	189	170	170
query20	127	129	115	115
query21	220	128	108	108
query22	4133	4136	4011	4011
query23	33976	33274	33435	33274
query24	8203	2359	2364	2359
query25	533	461	404	404
query26	1233	270	154	154
query27	2747	499	344	344
query28	4405	2198	2179	2179
query29	757	604	461	461
query30	283	224	192	192
query31	879	764	713	713
query32	79	73	73	73
query33	549	369	347	347
query34	778	842	509	509
query35	813	825	741	741
query36	949	1023	921	921
query37	115	112	93	93
query38	4089	4059	3972	3972
query39	1494	1406	1418	1406
query40	216	125	118	118
query41	58	56	54	54
query42	120	110	111	110
query43	490	500	460	460
query44	1380	854	849	849
query45	180	163	162	162
query46	846	1014	636	636
query47	1738	1778	1713	1713
query48	391	409	324	324
query49	703	486	383	383
query50	682	717	414	414
query51	4071	4150	4111	4111
query52	114	114	102	102
query53	235	267	191	191
query54	577	596	521	521
query55	90	84	90	84
query56	319	305	308	305
query57	1197	1181	1110	1110
query58	281	261	269	261
query59	2600	2677	2572	2572
query60	349	335	312	312
query61	127	129	118	118
query62	799	747	660	660
query63	261	191	189	189
query64	4344	1006	691	691
query65	4287	4185	4201	4185
query66	1137	427	350	350
query67	15494	15568	14972	14972
query68	8528	929	577	577
query69	483	322	273	273
query70	1210	1147	1100	1100
query71	450	326	312	312
query72	5320	4744	4826	4744
query73	749	641	357	357
query74	9014	9180	8856	8856
query75	3825	3090	2649	2649
query76	3649	1142	754	754
query77	795	398	316	316
query78	9510	9762	8786	8786
query79	1344	844	601	601
query80	602	537	478	478
query81	458	258	216	216
query82	200	141	108	108
query83	267	249	237	237
query84	244	95	93	93
query85	759	392	334	334
query86	337	296	291	291
query87	4269	4325	4201	4201
query88	2834	2197	2193	2193
query89	378	315	287	287
query90	2050	218	224	218
query91	138	141	114	114
query92	81	72	66	66
query93	1101	1009	640	640
query94	672	398	302	302
query95	404	317	295	295
query96	480	584	278	278
query97	2590	2717	2564	2564
query98	241	216	210	210
query99	1429	1454	1286	1286
Total cold run time: 269637 ms
Total hot run time: 184423 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

ClickBench: Total hot run time: 32.92 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46054238b355e495bfe468edca43b9db1118157c, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.05
query3	0.24	0.08	0.07
query4	1.61	0.11	0.11
query5	0.42	0.40	0.40
query6	1.16	0.63	0.67
query7	0.04	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.53	0.52
query10	0.58	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.64	0.61	0.60
query14	0.81	0.84	0.82
query15	0.89	0.86	0.87
query16	0.39	0.40	0.40
query17	1.06	1.03	1.02
query18	0.21	0.19	0.19
query19	1.91	1.91	1.88
query20	0.02	0.01	0.01
query21	15.37	0.94	0.59
query22	0.75	1.17	0.73
query23	14.92	1.40	0.64
query24	7.13	0.85	0.79
query25	0.54	0.20	0.18
query26	0.62	0.16	0.13
query27	0.06	0.05	0.05
query28	10.09	0.94	0.43
query29	12.64	3.95	3.35
query30	3.07	3.03	2.95
query31	2.82	0.58	0.39
query32	3.23	0.55	0.50
query33	3.11	3.06	3.09
query34	15.95	5.44	4.89
query35	4.95	4.95	5.01
query36	0.68	0.50	0.49
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.15
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.03	0.04	0.04
Total cold run time: 107.47 s
Total hot run time: 32.92 s

starocean999
starocean999 previously approved these changes Aug 19, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 19, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 19, 2025

PR approved by anyone and no changes requested.

@924060929 924060929 changed the title [opt](nereids) optimize parallel of insert command when olap table has empty partitions [opt](nereids) optimize parallel of insert command when olap table have auto partitions Aug 19, 2025
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 19, 2025
@924060929
Copy link
Contributor Author

924060929 commented Aug 19, 2025

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Aug 19, 2025

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-H: Total hot run time: 33830 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24ebe9dcb50829c47743dd7f52933d0d437b17ed, data reload: false

------ Round 1 ----------------------------------
q1	17609	5150	5071	5071
q2	1912	288	179	179
q3	10320	1270	735	735
q4	10219	998	505	505
q5	7527	2394	2310	2310
q6	173	156	128	128
q7	904	740	600	600
q8	9301	1331	1050	1050
q9	6886	5154	5203	5154
q10	6941	2373	1978	1978
q11	482	299	278	278
q12	345	350	212	212
q13	17787	3588	3057	3057
q14	236	240	205	205
q15	562	493	475	475
q16	429	415	387	387
q17	593	854	364	364
q18	7407	7048	7002	7002
q19	1099	951	562	562
q20	339	347	218	218
q21	3968	2631	2372	2372
q22	1112	1045	988	988
Total cold run time: 106151 ms
Total hot run time: 33830 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5161	5086	5131	5086
q2	240	317	220	220
q3	2144	2679	2237	2237
q4	1357	1748	1366	1366
q5	4208	4419	4460	4419
q6	226	171	126	126
q7	2012	1986	1726	1726
q8	2758	2610	2526	2526
q9	7227	7296	7376	7296
q10	3073	3370	2877	2877
q11	588	576	480	480
q12	691	823	601	601
q13	3487	3868	3469	3469
q14	275	305	286	286
q15	519	469	477	469
q16	458	502	443	443
q17	1169	1541	1431	1431
q18	7980	7716	7579	7579
q19	847	886	873	873
q20	1998	1996	1827	1827
q21	4708	4289	4319	4289
q22	1103	1050	1002	1002
Total cold run time: 52229 ms
Total hot run time: 50628 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-DS: Total hot run time: 184134 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24ebe9dcb50829c47743dd7f52933d0d437b17ed, data reload: false

query1	984	390	404	390
query2	6544	1749	1727	1727
query3	6739	220	221	220
query4	25914	23275	22986	22986
query5	4342	606	499	499
query6	306	219	238	219
query7	4621	495	286	286
query8	262	226	214	214
query9	8565	2866	2846	2846
query10	466	322	271	271
query11	15695	15071	14707	14707
query12	161	108	106	106
query13	1645	536	429	429
query14	9050	5723	5690	5690
query15	198	180	162	162
query16	7246	614	472	472
query17	1213	693	564	564
query18	1977	403	315	315
query19	182	184	160	160
query20	123	122	111	111
query21	212	121	111	111
query22	4184	4225	4136	4136
query23	34056	33284	33389	33284
query24	8181	2394	2360	2360
query25	562	496	427	427
query26	1242	270	164	164
query27	2760	507	352	352
query28	4436	2232	2187	2187
query29	784	577	475	475
query30	288	231	187	187
query31	912	796	710	710
query32	83	75	77	75
query33	567	363	354	354
query34	790	845	525	525
query35	810	827	738	738
query36	984	1016	930	930
query37	132	113	88	88
query38	4033	4018	3886	3886
query39	1496	1427	1437	1427
query40	213	120	113	113
query41	62	54	53	53
query42	119	102	105	102
query43	519	506	484	484
query44	1369	856	848	848
query45	174	172	167	167
query46	835	1006	644	644
query47	1763	1800	1713	1713
query48	392	418	322	322
query49	722	551	380	380
query50	625	674	400	400
query51	4082	4138	4087	4087
query52	115	114	102	102
query53	229	276	194	194
query54	599	610	522	522
query55	91	90	80	80
query56	309	320	291	291
query57	1174	1195	1108	1108
query58	279	266	289	266
query59	2664	2662	2563	2563
query60	346	342	317	317
query61	130	136	120	120
query62	805	774	661	661
query63	230	198	194	194
query64	4324	1000	687	687
query65	4263	4156	4211	4156
query66	1143	435	315	315
query67	15612	15304	15063	15063
query68	7872	954	570	570
query69	480	321	291	291
query70	1196	1105	1078	1078
query71	450	339	328	328
query72	5721	4747	4864	4747
query73	694	607	347	347
query74	9097	9105	8553	8553
query75	3772	3036	2609	2609
query76	3652	1124	715	715
query77	803	388	326	326
query78	9775	9530	8908	8908
query79	2791	787	601	601
query80	606	528	483	483
query81	497	259	212	212
query82	460	137	102	102
query83	274	246	228	228
query84	289	99	82	82
query85	865	383	332	332
query86	387	317	304	304
query87	4354	4251	4116	4116
query88	3618	2181	2184	2181
query89	390	311	281	281
query90	1878	216	219	216
query91	142	137	113	113
query92	91	70	68	68
query93	2130	972	635	635
query94	675	407	307	307
query95	466	311	303	303
query96	487	570	274	274
query97	2668	2673	2584	2584
query98	243	216	214	214
query99	1444	1406	1284	1284
Total cold run time: 274200 ms
Total hot run time: 184134 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

ClickBench: Total hot run time: 32.58 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24ebe9dcb50829c47743dd7f52933d0d437b17ed, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.08	0.08
query4	1.62	0.12	0.11
query5	0.42	0.42	0.40
query6	1.17	0.62	0.66
query7	0.02	0.02	0.02
query8	0.06	0.04	0.03
query9	0.61	0.52	0.53
query10	0.59	0.57	0.59
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.64	0.61	0.60
query14	0.79	0.83	0.80
query15	0.89	0.85	0.87
query16	0.37	0.40	0.39
query17	1.03	1.02	1.02
query18	0.22	0.19	0.19
query19	1.92	1.84	1.78
query20	0.02	0.01	0.01
query21	15.42	0.96	0.57
query22	0.75	1.20	0.72
query23	14.86	1.33	0.61
query24	6.78	1.16	0.82
query25	0.49	0.16	0.16
query26	0.63	0.16	0.13
query27	0.06	0.05	0.05
query28	9.79	0.91	0.42
query29	12.56	3.88	3.22
query30	3.13	3.04	3.06
query31	2.83	0.57	0.39
query32	3.22	0.54	0.48
query33	3.10	3.13	3.07
query34	16.04	5.44	4.86
query35	4.87	4.91	4.94
query36	0.69	0.50	0.48
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.03	0.03	0.02
query40	0.17	0.15	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.03	0.04	0.04
Total cold run time: 106.76 s
Total hot run time: 32.58 s

@924060929 924060929 merged commit 3f3e0ae into apache:master Aug 20, 2025
27 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Aug 20, 2025
@924060929
[opt](nereids) optimize parallel of insert command when olap table ha…
a0abbba 
…ve auto partitions (#54983)

optimize parallel of insert command when olap table have auto partitions
@github-actions github-actions bot mentioned this pull request Aug 20, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Aug 20, 2025
@924060929
[opt](nereids) optimize parallel of insert command when olap table ha…
ef62cda 
…ve auto partitions (#54983)

optimize parallel of insert command when olap table have auto partitions
@github-actions github-actions bot mentioned this pull request Aug 20, 2025
Open
github-actions bot pushed a commit that referenced this pull request Aug 20, 2025
@924060929
[opt](nereids) optimize parallel of insert command when olap table ha…
31d57c7 
…ve auto partitions (#54983)

optimize parallel of insert command when olap table have auto partitions
@github-actions github-actions bot mentioned this pull request Aug 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 4, 2025
@github-actions @924060929
branch-3.1: [opt](nereids) optimize parallel of insert command when o…
2208ead 
…lap table have auto partitions #54983 (#55030)

Cherry-picked from #54983

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
dataroaring pushed a commit that referenced this pull request Sep 5, 2025
@github-actions @924060929
branch-3.0: [opt](nereids) optimize parallel of insert command when o…
07ef370 
…lap table have auto partitions #54983 (#55028)

Cherry-picked from #54983

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
@924060929 924060929 deleted the opt-insert-auto-partition branch December 25, 2025 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.9-merged dev/3.1.1-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@924060929 @Thearas @doris-robot @starocean999 @morrySnow @dataroaring

Comments