Skip to content

[feat](function) Add function of deduplicate_map#58403

Merged
yiguolei merged 1 commit intoapache:masterfrom
mrhhsg:deduplicate_map
Nov 28, 2025
Merged

[feat](function) Add function of deduplicate_map#58403
yiguolei merged 1 commit intoapache:masterfrom
mrhhsg:deduplicate_map

Conversation

@mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Nov 26, 2025 •
edited
Loading

What problem does this PR solve?

Doc: apache/doris-website#3140
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@mrhhsg mrhhsg requested a review from zclllyybb as a code owner November 26, 2025 08:09
@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 26, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

zclllyybb
zclllyybb reviewed Nov 26, 2025
View reviewed changes
Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no regression cases?

@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 26, 2025 •
edited
Loading

@zclllyybb

no regression cases?

Normally, it is difficult to generate a map that contains duplicate keys.

@doris-robot
Copy link

doris-robot commented Nov 26, 2025

TPC-H: Total hot run time: 34453 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f53260c02509e1ccfdeb44b37c243d4c01146920, data reload: false

------ Round 1 ----------------------------------
q1	17617	5130	4962	4962
q2	2018	343	214	214
q3	10236	1328	745	745
q4	10238	974	372	372
q5	7566	2337	2432	2337
q6	187	169	141	141
q7	935	812	647	647
q8	9366	1361	1164	1164
q9	7174	5322	5244	5244
q10	6926	2252	1853	1853
q11	529	297	280	280
q12	368	363	224	224
q13	17778	3673	3018	3018
q14	245	246	219	219
q15	612	528	516	516
q16	1041	1025	991	991
q17	589	873	367	367
q18	7844	7082	7185	7082
q19	1208	958	554	554
q20	358	346	234	234
q21	3680	3211	2328	2328
q22	1048	1019	961	961
Total cold run time: 107563 ms
Total hot run time: 34453 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5039	4980	4959	4959
q2	333	400	312	312
q3	2152	2664	2271	2271
q4	1343	1782	1342	1342
q5	4247	4511	4543	4511
q6	216	179	134	134
q7	2100	2086	1763	1763
q8	2802	2646	2545	2545
q9	7460	7554	7545	7545
q10	3028	3337	2850	2850
q11	597	521	508	508
q12	720	747	646	646
q13	3734	3901	3277	3277
q14	293	325	322	322
q15	546	507	511	507
q16	1118	1130	1077	1077
q17	1183	1575	1441	1441
q18	8127	7712	7410	7410
q19	817	851	871	851
q20	2024	2036	1818	1818
q21	4744	4230	4332	4230
q22	1111	1056	1009	1009
Total cold run time: 53734 ms
Total hot run time: 51328 ms

@doris-robot
Copy link

doris-robot commented Nov 26, 2025

TPC-DS: Total hot run time: 184045 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f53260c02509e1ccfdeb44b37c243d4c01146920, data reload: false

query1	1080	402	394	394
query2	6548	1622	1621	1621
query3	6757	221	219	219
query4	26421	22531	22831	22531
query5	4399	616	467	467
query6	341	234	218	218
query7	4637	501	294	294
query8	301	269	260	260
query9	8701	2569	2572	2569
query10	524	361	306	306
query11	15587	14852	14485	14485
query12	173	121	116	116
query13	1679	573	431	431
query14	10505	8903	8847	8847
query15	209	203	185	185
query16	7236	690	541	541
query17	1234	739	623	623
query18	1988	410	320	320
query19	205	200	176	176
query20	134	125	120	120
query21	215	135	113	113
query22	3843	3895	3907	3895
query23	32732	31909	31970	31909
query24	8412	2378	2369	2369
query25	606	503	466	466
query26	1231	273	163	163
query27	2764	483	342	342
query28	4384	2128	2130	2128
query29	836	620	481	481
query30	314	237	209	209
query31	860	714	625	625
query32	79	79	72	72
query33	588	383	330	330
query34	785	875	547	547
query35	816	855	753	753
query36	894	960	862	862
query37	123	110	83	83
query38	3381	3313	3192	3192
query39	1493	1419	1422	1419
query40	225	132	118	118
query41	67	65	61	61
query42	132	108	117	108
query43	458	466	425	425
query44	1245	766	757	757
query45	199	192	183	183
query46	886	993	661	661
query47	1674	1706	1670	1670
query48	390	450	324	324
query49	792	502	420	420
query50	662	683	420	420
query51	3829	4014	3817	3817
query52	109	109	111	109
query53	242	265	194	194
query54	298	286	269	269
query55	90	92	93	92
query56	335	325	321	321
query57	1178	1191	1117	1117
query58	290	269	273	269
query59	2320	2457	2385	2385
query60	359	346	327	327
query61	157	154	155	154
query62	785	713	659	659
query63	230	190	188	188
query64	4539	1169	1029	1029
query65	4072	3977	3952	3952
query66	1186	450	363	363
query67	15143	14886	14815	14815
query68	4637	932	651	651
query69	558	347	322	322
query70	1300	1189	1261	1189
query71	436	324	319	319
query72	6113	5105	5088	5088
query73	640	582	345	345
query74	8661	8818	8538	8538
query75	3282	3333	2861	2861
query76	3244	1113	747	747
query77	489	398	308	308
query78	9571	9858	8946	8946
query79	1937	819	582	582
query80	681	564	480	480
query81	520	275	240	240
query82	239	157	130	130
query83	262	261	249	249
query84	265	107	103	103
query85	886	480	441	441
query86	380	314	296	296
query87	3552	3527	3415	3415
query88	2818	2256	2226	2226
query89	382	324	281	281
query90	1994	232	215	215
query91	172	166	135	135
query92	87	66	63	63
query93	1793	1004	664	664
query94	723	424	334	334
query95	502	402	388	388
query96	496	562	274	274
query97	2870	2981	2854	2854
query98	236	212	217	212
query99	1296	1375	1261	1261
Total cold run time: 265929 ms
Total hot run time: 184045 ms

@doris-robot
Copy link

doris-robot commented Nov 26, 2025

ClickBench: Total hot run time: 27.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f53260c02509e1ccfdeb44b37c243d4c01146920, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.27	0.26	0.25
query6	1.19	0.64	0.62
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.53	0.51
query10	0.57	0.58	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.60	0.60
query14	0.99	1.01	0.99
query15	0.84	0.83	0.84
query16	0.40	0.38	0.41
query17	1.03	1.01	1.00
query18	0.21	0.20	0.20
query19	1.92	1.78	1.80
query20	0.01	0.02	0.01
query21	15.48	0.21	0.13
query22	4.97	0.06	0.05
query23	15.69	0.26	0.11
query24	2.63	1.05	0.40
query25	0.08	0.07	0.05
query26	0.15	0.13	0.12
query27	0.07	0.05	0.06
query28	4.38	1.14	0.92
query29	12.57	3.84	3.26
query30	0.27	0.13	0.13
query31	2.82	0.60	0.39
query32	3.22	0.54	0.46
query33	2.96	3.08	3.05
query34	15.87	5.21	4.53
query35	4.57	4.57	4.62
query36	0.68	0.49	0.52
query37	0.09	0.06	0.07
query38	0.07	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.97 s
Total hot run time: 27.35 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 26, 2025

FE Regression Coverage Report

Increment line coverage 10.00% (1/10) 🎉
Increment coverage report
Complete coverage report

@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 27, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-H: Total hot run time: 34522 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2fd46a7baabb04a5b41dadfbab05c9f2e150944b, data reload: false

------ Round 1 ----------------------------------
q1	17604	5095	4948	4948
q2	2065	323	213	213
q3	10250	1294	735	735
q4	10225	971	368	368
q5	7554	2388	2377	2377
q6	194	180	144	144
q7	954	777	628	628
q8	9333	1369	1171	1171
q9	7219	5395	5362	5362
q10	6914	2217	1802	1802
q11	502	297	284	284
q12	359	366	234	234
q13	17766	3664	3010	3010
q14	244	262	211	211
q15	580	523	510	510
q16	1013	1021	962	962
q17	602	882	363	363
q18	7323	7254	7098	7098
q19	1101	954	553	553
q20	360	343	227	227
q21	3931	3247	2361	2361
q22	1057	1021	961	961
Total cold run time: 107150 ms
Total hot run time: 34522 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5043	4990	4995	4990
q2	324	406	319	319
q3	2188	2662	2372	2372
q4	1370	1760	1342	1342
q5	4259	4530	4549	4530
q6	217	188	136	136
q7	2085	1958	1850	1850
q8	2690	2644	2556	2556
q9	7511	7599	7441	7441
q10	3122	3242	2897	2897
q11	587	523	511	511
q12	689	772	647	647
q13	3570	3929	3278	3278
q14	310	318	299	299
q15	552	500	499	499
q16	1084	1147	1133	1133
q17	1180	1548	1408	1408
q18	8076	7652	7613	7613
q19	850	849	997	849
q20	2022	2094	1918	1918
q21	4818	4325	4288	4288
q22	1113	1039	1002	1002
Total cold run time: 53660 ms
Total hot run time: 51878 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE UT Coverage Report

Increment line coverage 10.00% (1/10) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-DS: Total hot run time: 184708 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2fd46a7baabb04a5b41dadfbab05c9f2e150944b, data reload: false

query1	1049	424	400	400
query2	6591	1592	1585	1585
query3	6779	222	218	218
query4	25851	22570	22535	22535
query5	4406	638	462	462
query6	332	255	228	228
query7	4666	494	298	298
query8	292	258	240	240
query9	8729	2597	2571	2571
query10	521	376	301	301
query11	15391	14768	14636	14636
query12	177	119	113	113
query13	1680	574	467	467
query14	10730	8987	8893	8893
query15	214	208	185	185
query16	7359	666	522	522
query17	1257	773	641	641
query18	2029	440	346	346
query19	211	211	189	189
query20	133	135	122	122
query21	219	137	114	114
query22	4194	4210	4088	4088
query23	33167	32468	32034	32034
query24	8466	2410	2397	2397
query25	628	548	502	502
query26	1243	287	170	170
query27	2738	498	356	356
query28	4383	2139	2127	2127
query29	865	644	535	535
query30	315	243	223	223
query31	859	727	675	675
query32	82	72	73	72
query33	604	418	330	330
query34	790	873	542	542
query35	828	853	760	760
query36	923	967	887	887
query37	124	110	87	87
query38	3343	3361	3322	3322
query39	1506	1511	1410	1410
query40	224	129	119	119
query41	67	62	64	62
query42	128	117	111	111
query43	462	445	447	445
query44	1238	764	757	757
query45	196	193	188	188
query46	886	974	632	632
query47	1729	1754	1704	1704
query48	383	430	325	325
query49	763	509	408	408
query50	651	667	402	402
query51	3877	3895	3884	3884
query52	111	111	102	102
query53	241	262	193	193
query54	298	282	282	282
query55	99	97	95	95
query56	349	327	303	303
query57	1195	1158	1118	1118
query58	279	276	276	276
query59	2339	2463	2408	2408
query60	353	346	370	346
query61	159	159	157	157
query62	779	721	647	647
query63	218	191	191	191
query64	4496	1207	895	895
query65	4039	3957	3963	3957
query66	1202	447	337	337
query67	15179	15325	14763	14763
query68	8307	954	621	621
query69	514	350	298	298
query70	1308	1223	1233	1223
query71	474	337	308	308
query72	5644	4962	4873	4873
query73	687	587	342	342
query74	8533	8885	8647	8647
query75	3972	3326	2868	2868
query76	3720	1131	717	717
query77	820	393	325	325
query78	9520	9933	8931	8931
query79	1979	826	585	585
query80	638	590	488	488
query81	501	271	246	246
query82	423	166	131	131
query83	273	262	248	248
query84	254	111	100	100
query85	932	486	452	452
query86	345	295	290	290
query87	3531	3593	3383	3383
query88	3250	2241	2232	2232
query89	379	324	290	290
query90	1937	223	212	212
query91	175	166	139	139
query92	79	69	64	64
query93	1206	989	657	657
query94	727	450	326	326
query95	495	405	387	387
query96	499	565	280	280
query97	2924	2994	2850	2850
query98	249	219	225	219
query99	1653	1384	1286	1286
Total cold run time: 271562 ms
Total hot run time: 184708 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

ClickBench: Total hot run time: 27.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2fd46a7baabb04a5b41dadfbab05c9f2e150944b, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.04
query3	0.25	0.08	0.09
query4	1.60	0.11	0.11
query5	0.28	0.26	0.27
query6	1.18	0.64	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.60	0.53	0.51
query10	0.57	0.57	0.58
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.62
query14	0.99	1.01	0.99
query15	0.85	0.83	0.83
query16	0.39	0.40	0.41
query17	1.01	1.03	1.02
query18	0.22	0.20	0.23
query19	1.90	1.87	1.78
query20	0.01	0.02	0.02
query21	15.45	0.20	0.14
query22	5.03	0.07	0.05
query23	15.67	0.25	0.11
query24	2.88	1.33	0.88
query25	0.07	0.07	0.06
query26	0.14	0.13	0.14
query27	0.06	0.05	0.05
query28	5.21	1.15	0.92
query29	12.61	3.99	3.25
query30	0.28	0.14	0.11
query31	2.82	0.60	0.40
query32	3.24	0.54	0.47
query33	2.99	3.10	3.05
query34	15.83	5.14	4.53
query35	4.55	4.54	4.54
query36	0.68	0.50	0.48
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 99.1 s
Total hot run time: 27.94 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE UT Coverage Report

Increment line coverage 10.00% (1/10) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE Regression Coverage Report

Increment line coverage 60.00% (6/10) 🎉
Increment coverage report
Complete coverage report

@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 27, 2025

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2025

Possible file(s) that should be tracked in LFS detected: 🚨

The following file(s) exceeds the file size limit: 1048576 bytes, as set in the .yml configuration files:

  • regression-test/data/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.out
  • regression-test/data/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.out.txt

Consider using git-lfs to manage large files.

@github-actions github-actions bot added the lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request label Nov 27, 2025
@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-H: Total hot run time: 34002 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7067f61dc034816971be8518d31d06fc95ddb6a3, data reload: false

------ Round 1 ----------------------------------
q1	17652	5094	4965	4965
q2	2019	352	205	205
q3	10236	1291	712	712
q4	10230	897	362	362
q5	7487	2373	2312	2312
q6	188	172	143	143
q7	928	777	629	629
q8	9345	1336	1079	1079
q9	7086	5325	5257	5257
q10	6864	2227	1810	1810
q11	487	295	280	280
q12	343	358	223	223
q13	17792	3679	3010	3010
q14	235	238	217	217
q15	575	520	511	511
q16	1010	1010	960	960
q17	577	857	360	360
q18	7779	7132	6973	6973
q19	1083	944	542	542
q20	350	345	230	230
q21	3697	2521	2260	2260
q22	1048	979	962	962
Total cold run time: 107011 ms
Total hot run time: 34002 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4975	4998	4981	4981
q2	330	402	312	312
q3	2171	2731	2277	2277
q4	1378	1789	1321	1321
q5	4215	4354	4570	4354
q6	215	181	133	133
q7	2056	1962	1885	1885
q8	2705	2621	2534	2534
q9	7521	7473	7551	7473
q10	3046	3356	2838	2838
q11	586	538	489	489
q12	695	759	673	673
q13	3566	3948	3424	3424
q14	304	316	291	291
q15	549	511	504	504
q16	1080	1185	1077	1077
q17	1176	1560	1397	1397
q18	7842	7595	7564	7564
q19	778	767	875	767
q20	1915	1951	1811	1811
q21	4711	4279	4229	4229
q22	1114	1041	1014	1014
Total cold run time: 52928 ms
Total hot run time: 51348 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-DS: Total hot run time: 185583 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7067f61dc034816971be8518d31d06fc95ddb6a3, data reload: false

query1	1072	408	393	393
query2	6580	1598	1619	1598
query3	6753	230	223	223
query4	26571	23117	22806	22806
query5	4476	675	480	480
query6	340	251	228	228
query7	4656	507	296	296
query8	345	262	253	253
query9	8709	2554	2592	2554
query10	526	362	315	315
query11	15278	14839	14659	14659
query12	182	128	119	119
query13	1710	587	450	450
query14	10265	9119	9248	9119
query15	222	204	187	187
query16	7511	716	542	542
query17	1277	766	664	664
query18	2018	449	343	343
query19	215	215	197	197
query20	143	128	129	128
query21	221	135	117	117
query22	3900	4044	3989	3989
query23	33187	31909	32294	31909
query24	8507	2494	2535	2494
query25	627	545	490	490
query26	1243	275	170	170
query27	2729	499	364	364
query28	4374	2173	2160	2160
query29	865	641	521	521
query30	322	247	217	217
query31	853	736	619	619
query32	85	77	76	76
query33	609	391	348	348
query34	795	932	545	545
query35	822	829	761	761
query36	938	969	880	880
query37	129	111	90	90
query38	3356	3361	3301	3301
query39	1510	1423	1439	1423
query40	229	131	121	121
query41	67	66	66	66
query42	125	115	114	114
query43	472	472	458	458
query44	1273	769	749	749
query45	202	196	191	191
query46	883	1001	661	661
query47	1675	1721	1647	1647
query48	395	444	327	327
query49	798	507	424	424
query50	689	711	416	416
query51	3931	3960	3838	3838
query52	114	117	106	106
query53	246	269	199	199
query54	309	288	274	274
query55	93	96	95	95
query56	334	351	314	314
query57	1143	1160	1100	1100
query58	288	290	274	274
query59	2389	2494	2433	2433
query60	358	361	340	340
query61	158	162	164	162
query62	805	706	670	670
query63	240	201	205	201
query64	4516	1248	877	877
query65	4085	3998	4003	3998
query66	1174	448	336	336
query67	15195	15019	14733	14733
query68	8216	920	627	627
query69	524	349	306	306
query70	1215	1239	1220	1220
query71	479	343	318	318
query72	6136	4934	4858	4858
query73	673	577	345	345
query74	8594	8867	8855	8855
query75	3963	3359	2886	2886
query76	3639	1139	737	737
query77	821	416	334	334
query78	9622	9620	8903	8903
query79	2280	836	588	588
query80	651	570	525	525
query81	517	271	244	244
query82	469	158	130	130
query83	264	260	252	252
query84	270	117	103	103
query85	930	478	453	453
query86	389	291	302	291
query87	3498	3542	3412	3412
query88	3729	2257	2281	2257
query89	392	311	291	291
query90	1843	217	211	211
query91	174	175	144	144
query92	101	68	67	67
query93	1862	981	656	656
query94	761	459	340	340
query95	497	396	397	396
query96	488	569	285	285
query97	2919	2960	2877	2877
query98	244	214	208	208
query99	1653	1361	1312	1312
Total cold run time: 273920 ms
Total hot run time: 185583 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

ClickBench: Total hot run time: 27.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7067f61dc034816971be8518d31d06fc95ddb6a3, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.04
query3	0.26	0.09	0.08
query4	1.61	0.11	0.11
query5	0.26	0.25	0.26
query6	1.18	0.66	0.64
query7	0.03	0.03	0.02
query8	0.06	0.05	0.04
query9	0.59	0.52	0.51
query10	0.58	0.57	0.58
query11	0.16	0.12	0.12
query12	0.15	0.11	0.12
query13	0.63	0.61	0.60
query14	1.00	1.01	1.01
query15	0.86	0.82	0.84
query16	0.40	0.40	0.40
query17	1.06	1.02	1.04
query18	0.22	0.19	0.19
query19	1.99	1.80	1.82
query20	0.02	0.02	0.01
query21	15.44	0.19	0.13
query22	5.01	0.07	0.05
query23	15.68	0.26	0.11
query24	2.72	0.45	0.59
query25	0.08	0.06	0.06
query26	0.15	0.14	0.13
query27	0.07	0.06	0.06
query28	4.93	1.16	0.92
query29	12.60	3.86	3.18
query30	0.28	0.14	0.11
query31	2.82	0.60	0.38
query32	3.34	0.55	0.47
query33	3.02	3.10	3.07
query34	15.89	5.16	4.59
query35	4.60	4.57	4.54
query36	0.65	0.51	0.48
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 99.04 s
Total hot run time: 27.48 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE Regression Coverage Report

Increment line coverage 60.00% (6/10) 🎉
Increment coverage report
Complete coverage report

@mrhhsg
Copy link
Member Author

mrhhsg commented Nov 27, 2025

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2025

Possible file(s) that should be tracked in LFS detected: 🚨

The following file(s) exceeds the file size limit: 1048576 bytes, as set in the .yml configuration files:

  • regression-test/data/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.out
  • regression-test/data/datatype_p0/nested_types/base_cases/one_level_nestedtypes_with_s3data.out.txt

Consider using git-lfs to manage large files.

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-H: Total hot run time: 34307 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d5498092e9b7ffeecfc003a3bb66df0805e715db, data reload: false

------ Round 1 ----------------------------------
q1	17631	5173	4911	4911
q2	2045	358	210	210
q3	10217	1299	736	736
q4	10226	956	377	377
q5	7555	2380	2294	2294
q6	186	174	137	137
q7	935	758	643	643
q8	9358	1321	1031	1031
q9	6984	5226	5304	5226
q10	6823	2208	1807	1807
q11	502	305	283	283
q12	340	370	227	227
q13	17777	3605	3007	3007
q14	236	233	218	218
q15	576	527	518	518
q16	1001	1009	979	979
q17	577	807	447	447
q18	7428	7048	7089	7048
q19	1034	960	573	573
q20	335	338	230	230
q21	2937	3107	2462	2462
q22	1014	1009	943	943
Total cold run time: 105717 ms
Total hot run time: 34307 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4940	4899	4895	4895
q2	325	398	320	320
q3	2115	2639	2246	2246
q4	1320	1751	1291	1291
q5	4176	4202	4584	4202
q6	215	172	135	135
q7	2043	2025	1866	1866
q8	2613	2436	2434	2434
q9	7574	7533	7519	7519
q10	3007	3278	2770	2770
q11	592	529	496	496
q12	708	784	594	594
q13	3548	3975	3312	3312
q14	292	290	289	289
q15	539	503	509	503
q16	1104	1138	1093	1093
q17	1140	1364	1421	1364
q18	8062	7723	7514	7514
q19	818	799	834	799
q20	1954	2110	1879	1879
q21	4952	4311	4143	4143
q22	1064	1041	1002	1002
Total cold run time: 53101 ms
Total hot run time: 50666 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-DS: Total hot run time: 184704 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d5498092e9b7ffeecfc003a3bb66df0805e715db, data reload: false

query1	1031	450	392	392
query2	6576	1577	1591	1577
query3	6757	227	221	221
query4	25418	23027	22852	22852
query5	4428	627	505	505
query6	338	236	229	229
query7	4648	515	311	311
query8	312	243	240	240
query9	8727	2602	2616	2602
query10	502	352	322	322
query11	15205	14704	14559	14559
query12	183	117	113	113
query13	1714	587	466	466
query14	10732	8835	8766	8766
query15	217	204	187	187
query16	7355	719	538	538
query17	1259	834	617	617
query18	2003	419	334	334
query19	206	203	176	176
query20	128	134	122	122
query21	219	136	114	114
query22	4004	4178	3913	3913
query23	32914	32058	32034	32034
query24	8467	2390	2411	2390
query25	610	528	449	449
query26	1237	276	170	170
query27	2735	490	342	342
query28	4330	2180	2153	2153
query29	872	612	492	492
query30	319	242	208	208
query31	821	703	620	620
query32	86	77	72	72
query33	601	385	341	341
query34	819	867	537	537
query35	801	841	739	739
query36	874	930	851	851
query37	123	112	92	92
query38	3375	3293	3249	3249
query39	1487	1408	1426	1408
query40	234	130	123	123
query41	71	63	68	63
query42	129	109	112	109
query43	442	451	430	430
query44	1272	786	756	756
query45	197	185	186	185
query46	871	1007	650	650
query47	1716	1731	1661	1661
query48	394	431	333	333
query49	782	508	440	440
query50	649	678	403	403
query51	4035	3867	3850	3850
query52	113	117	109	109
query53	240	267	196	196
query54	306	296	272	272
query55	96	96	91	91
query56	330	351	320	320
query57	1139	1172	1099	1099
query58	287	271	279	271
query59	2373	2535	2325	2325
query60	362	346	338	338
query61	163	157	158	157
query62	788	719	647	647
query63	225	194	188	188
query64	4697	1330	1047	1047
query65	4062	3939	3961	3939
query66	1192	473	358	358
query67	15331	15197	15113	15113
query68	8296	956	632	632
query69	532	357	326	326
query70	1283	1308	1147	1147
query71	427	342	333	333
query72	6014	4951	4897	4897
query73	652	577	341	341
query74	8812	8845	8740	8740
query75	3308	3287	2744	2744
query76	3358	1121	707	707
query77	547	407	307	307
query78	9679	9653	8858	8858
query79	2526	823	580	580
query80	666	591	496	496
query81	531	271	239	239
query82	303	162	130	130
query83	274	263	253	253
query84	264	117	103	103
query85	878	511	439	439
query86	384	308	288	288
query87	3456	3468	3369	3369
query88	4264	2237	2277	2237
query89	395	342	298	298
query90	1987	217	219	217
query91	178	165	141	141
query92	84	72	66	66
query93	2461	1007	657	657
query94	723	467	344	344
query95	517	410	402	402
query96	513	554	280	280
query97	2912	2951	2862	2862
query98	243	217	212	212
query99	1294	1384	1283	1283
Total cold run time: 272690 ms
Total hot run time: 184704 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

ClickBench: Total hot run time: 27.38 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d5498092e9b7ffeecfc003a3bb66df0805e715db, data reload: false

query1	0.06	0.05	0.06
query2	0.11	0.04	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.11
query5	0.27	0.26	0.26
query6	1.15	0.65	0.64
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.58	0.51	0.52
query10	0.56	0.56	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.61	0.61
query14	0.99	0.99	0.99
query15	0.81	0.82	0.79
query16	0.39	0.38	0.39
query17	0.97	0.99	0.97
query18	0.24	0.21	0.22
query19	1.83	1.77	1.79
query20	0.01	0.01	0.01
query21	15.47	0.26	0.14
query22	5.10	0.05	0.05
query23	16.14	0.26	0.10
query24	2.36	0.32	0.50
query25	0.06	0.06	0.08
query26	0.14	0.14	0.14
query27	0.09	0.05	0.06
query28	4.37	1.23	1.02
query29	12.60	3.87	3.25
query30	0.29	0.14	0.12
query31	2.81	0.63	0.40
query32	3.23	0.56	0.47
query33	3.11	3.08	3.17
query34	16.87	5.17	4.53
query35	4.61	4.52	4.62
query36	0.64	0.50	0.49
query37	0.11	0.07	0.07
query38	0.06	0.05	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.03	0.04
query42	0.04	0.04	0.03
query43	0.04	0.03	0.04
Total cold run time: 99.3 s
Total hot run time: 27.38 s

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

BE UT Coverage Report

Increment line coverage 94.74% (18/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.90% (18511/34991)
Line Coverage 38.44% (169540/441065)
Region Coverage 33.23% (131568/395966)
Branch Coverage 34.19% (56682/165769)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (19/19) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.76% (24620/34309)
Line Coverage 58.34% (257084/440671)
Region Coverage 53.53% (214526/400776)
Branch Coverage 55.04% (91730/166664)

@yiguolei yiguolei added usercase Important user case type label dev/4.0.x labels Nov 28, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 28, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 28, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 28, 2025

PR approved by anyone and no changes requested.

zclllyybb
zclllyybb approved these changes Nov 28, 2025
View reviewed changes
Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 7ae9d26 into apache:master Nov 28, 2025
26 of 30 checks passed
@mrhhsg mrhhsg deleted the deduplicate_map branch November 30, 2025 11:50
mrhhsg added a commit to mrhhsg/doris that referenced this pull request Nov 30, 2025
@mrhhsg
[feat](function) Add function of deduplicate_map (apache#58403)
cd5ac59 
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@mrhhsg mrhhsg mentioned this pull request Nov 30, 2025
Merged
29 tasks
yiguolei pushed a commit that referenced this pull request Dec 2, 2025
@mrhhsg
[feat](function) Add function of deduplicate_map (#58403) (#58537)
6a27e21 
Issue Number: close #xxx

PICK PR: #58403

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@yiguolei yiguolei mentioned this pull request Dec 2, 2025
Open
@mrhhsg mrhhsg mentioned this pull request Dec 2, 2025
Merged
8 tasks
yiguolei pushed a commit to apache/doris-website that referenced this pull request Dec 8, 2025
@mrhhsg
add doc of deduplicate_map (#3140)
5f17aab 
apache/doris#58403

## Versions 

- [x] dev
- [ ] 4.x
- [ ] 3.x
- [ ] 2.1

## Languages

- [x] Chinese
- [x] English

## Docs Checklist

- [ ] Checked by AI
- [ ] Test Cases Built
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@mrhhsg @nagisa-kunhah
[feat](function) Add function of deduplicate_map (apache#58403)
cdd5faa 
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.2-merged lfs-detected! Warning Label for use when LFS is detected in the commits of a Pull Request reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mrhhsg @hello-stephen @doris-robot @yiguolei @zclllyybb

Comments