Skip to content

[Fix](SIMDJson) manually selecting the implementation for different CPU architecture#52732

Merged
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:simdjson-arch
Jul 16, 2025
Merged

[Fix](SIMDJson) manually selecting the implementation for different CPU architecture#52732
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:simdjson-arch

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Jul 3, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon
Copy link
Member Author

run buildall

@eldenmoon eldenmoon changed the title [Fix](SIMDJson) manually selecting the implementation for different C… [Fix](SIMDJson) manually selecting the implementation for different CPU architecture Jul 3, 2025
@eldenmoon eldenmoon added dev/2.1.x dev/3.0.x dev/3.1.x usercase Important user case type label labels Jul 3, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 33784 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99375850150f906eca1b997ec6bbdec14e166782, data reload: false

------ Round 1 ----------------------------------
q1	17613	5228	5133	5133
q2	1932	307	193	193
q3	10281	1307	718	718
q4	10220	1011	514	514
q5	7556	2292	2439	2292
q6	175	160	127	127
q7	872	763	596	596
q8	9318	1302	1051	1051
q9	6701	5065	5015	5015
q10	6883	2369	1932	1932
q11	477	297	295	295
q12	343	350	204	204
q13	17766	3685	3039	3039
q14	217	223	206	206
q15	540	469	494	469
q16	418	423	386	386
q17	626	870	375	375
q18	7524	7132	7243	7132
q19	1094	940	551	551
q20	338	360	226	226
q21	3909	3175	2369	2369
q22	1064	987	961	961
Total cold run time: 105867 ms
Total hot run time: 33784 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5200	5148	5191	5148
q2	245	336	222	222
q3	2198	2650	2248	2248
q4	1367	1808	1343	1343
q5	4224	4518	4637	4518
q6	219	165	133	133
q7	2009	1994	1851	1851
q8	2668	2614	2544	2544
q9	7310	7170	7248	7170
q10	3097	3283	2951	2951
q11	586	507	492	492
q12	723	787	620	620
q13	3551	3949	3379	3379
q14	283	298	274	274
q15	513	473	499	473
q16	451	551	460	460
q17	1163	1676	1366	1366
q18	7929	7695	7558	7558
q19	848	847	940	847
q20	2052	2073	1938	1938
q21	5027	4660	4585	4585
q22	1111	1100	1051	1051
Total cold run time: 52774 ms
Total hot run time: 51171 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185327 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99375850150f906eca1b997ec6bbdec14e166782, data reload: false

query1	1011	393	382	382
query2	6530	1729	1747	1729
query3	6745	214	214	214
query4	26605	23987	23337	23337
query5	4798	592	436	436
query6	307	232	207	207
query7	4631	500	282	282
query8	280	225	211	211
query9	8648	2627	2628	2627
query10	494	320	271	271
query11	15446	15144	14783	14783
query12	153	112	104	104
query13	1660	536	413	413
query14	9378	5649	5756	5649
query15	207	185	182	182
query16	7659	648	452	452
query17	1176	711	596	596
query18	2013	416	304	304
query19	200	204	166	166
query20	123	114	118	114
query21	215	128	114	114
query22	4620	4741	4646	4646
query23	34667	33499	32988	32988
query24	8398	2407	2360	2360
query25	500	457	379	379
query26	1224	262	143	143
query27	2724	504	342	342
query28	4247	2142	2124	2124
query29	727	547	433	433
query30	284	214	181	181
query31	896	820	760	760
query32	69	61	58	58
query33	539	373	303	303
query34	807	842	520	520
query35	801	815	747	747
query36	940	987	894	894
query37	100	91	71	71
query38	4309	4093	3994	3994
query39	1438	1403	1392	1392
query40	204	115	106	106
query41	53	54	49	49
query42	118	104	101	101
query43	500	504	482	482
query44	1318	821	805	805
query45	176	169	171	169
query46	850	1024	627	627
query47	1761	1794	1722	1722
query48	390	435	310	310
query49	737	491	367	367
query50	636	698	406	406
query51	4144	4139	4117	4117
query52	108	101	95	95
query53	227	256	191	191
query54	575	568	485	485
query55	88	81	82	81
query56	321	290	289	289
query57	1192	1187	1108	1108
query58	264	253	273	253
query59	2656	2727	2608	2608
query60	335	337	295	295
query61	123	116	118	116
query62	802	723	673	673
query63	220	183	187	183
query64	4194	981	653	653
query65	4276	4218	4163	4163
query66	1080	404	311	311
query67	15791	15749	15503	15503
query68	8711	889	517	517
query69	481	324	280	280
query70	1215	1134	1109	1109
query71	482	310	309	309
query72	5558	4689	4746	4689
query73	711	626	347	347
query74	9132	9043	8921	8921
query75	3956	3206	2647	2647
query76	3709	1158	711	711
query77	783	386	294	294
query78	10125	10095	9249	9249
query79	2453	818	577	577
query80	669	516	438	438
query81	460	272	220	220
query82	457	127	91	91
query83	276	241	228	228
query84	294	111	88	88
query85	788	360	316	316
query86	330	297	270	270
query87	4432	4481	4312	4312
query88	3459	2343	2306	2306
query89	389	321	279	279
query90	1935	205	213	205
query91	141	143	113	113
query92	76	61	55	55
query93	1436	965	589	589
query94	675	408	291	291
query95	379	294	286	286
query96	516	582	288	288
query97	2688	2751	2629	2629
query98	237	210	210	210
query99	1448	1469	1287	1287
Total cold run time: 277109 ms
Total hot run time: 185327 ms

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34204 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5228b9d402770cb094f2c20736f4d5377001c3fb, data reload: false

------ Round 1 ----------------------------------
q1	17587	5194	5075	5075
q2	1941	302	184	184
q3	10501	1291	720	720
q4	10305	1024	509	509
q5	8509	2389	2338	2338
q6	194	159	127	127
q7	864	737	609	609
q8	9320	1288	1165	1165
q9	6995	5128	5154	5128
q10	6905	2398	1952	1952
q11	491	304	285	285
q12	342	349	212	212
q13	17747	3630	3110	3110
q14	221	235	209	209
q15	546	478	486	478
q16	417	442	387	387
q17	624	862	393	393
q18	7864	7249	7107	7107
q19	1226	946	550	550
q20	332	339	250	250
q21	3949	3186	2422	2422
q22	1075	1046	994	994
Total cold run time: 107955 ms
Total hot run time: 34204 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5143	5123	5114	5114
q2	235	316	221	221
q3	2214	2681	2280	2280
q4	1407	1814	1302	1302
q5	4387	4679	4536	4536
q6	208	166	122	122
q7	2028	1977	1860	1860
q8	2642	2644	2574	2574
q9	7558	7231	7353	7231
q10	3110	3299	3105	3105
q11	562	531	485	485
q12	661	749	620	620
q13	3448	4101	3513	3513
q14	303	320	295	295
q15	522	505	475	475
q16	461	501	455	455
q17	1165	1582	1403	1403
q18	7927	7650	7701	7650
q19	828	818	990	818
q20	2040	2126	1909	1909
q21	5579	4412	4717	4412
q22	1147	1098	1107	1098
Total cold run time: 53575 ms
Total hot run time: 51478 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186650 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5228b9d402770cb094f2c20736f4d5377001c3fb, data reload: false

query1	1011	389	375	375
query2	6531	1722	1768	1722
query3	6737	216	209	209
query4	26365	24051	23394	23394
query5	4370	607	442	442
query6	312	220	214	214
query7	4623	511	291	291
query8	279	223	221	221
query9	8623	2647	2674	2647
query10	485	344	269	269
query11	15773	14981	14899	14899
query12	161	107	102	102
query13	1650	552	415	415
query14	9475	5875	5848	5848
query15	202	186	174	174
query16	7563	626	453	453
query17	1197	743	622	622
query18	2010	420	304	304
query19	208	204	166	166
query20	120	116	117	116
query21	218	123	109	109
query22	4083	4459	4733	4459
query23	35506	33909	33898	33898
query24	8422	2422	2412	2412
query25	532	479	385	385
query26	1224	263	148	148
query27	2724	507	345	345
query28	4283	2143	2109	2109
query29	738	563	434	434
query30	276	212	210	210
query31	911	821	743	743
query32	73	61	61	61
query33	543	366	305	305
query34	806	840	517	517
query35	765	823	749	749
query36	970	987	899	899
query37	107	98	72	72
query38	4090	4033	4083	4033
query39	1462	1387	1393	1387
query40	204	118	108	108
query41	53	52	49	49
query42	128	107	109	107
query43	500	488	469	469
query44	1368	824	812	812
query45	174	170	164	164
query46	855	1074	655	655
query47	1720	1757	1699	1699
query48	390	420	312	312
query49	738	468	382	382
query50	683	685	408	408
query51	4204	4181	4099	4099
query52	108	107	97	97
query53	236	260	188	188
query54	576	570	498	498
query55	86	82	80	80
query56	319	308	294	294
query57	1162	1204	1107	1107
query58	274	255	260	255
query59	2628	2762	2576	2576
query60	331	321	309	309
query61	119	117	123	117
query62	810	700	652	652
query63	241	192	186	186
query64	4268	985	721	721
query65	4261	4205	4173	4173
query66	1079	405	362	362
query67	15508	15529	15278	15278
query68	9181	899	522	522
query69	486	318	298	298
query70	1212	1153	1087	1087
query71	505	322	297	297
query72	5490	4780	4874	4780
query73	722	619	348	348
query74	9166	8787	8735	8735
query75	4333	3210	2678	2678
query76	3738	1167	753	753
query77	805	379	292	292
query78	10080	10162	9347	9347
query79	3370	792	570	570
query80	648	510	448	448
query81	465	258	230	230
query82	458	122	98	98
query83	284	242	227	227
query84	291	109	83	83
query85	793	352	348	348
query86	327	295	295	295
query87	4358	4436	4329	4329
query88	3417	2305	2300	2300
query89	432	322	282	282
query90	1993	221	213	213
query91	139	138	109	109
query92	70	60	56	56
query93	2754	939	585	585
query94	679	395	312	312
query95	381	296	283	283
query96	500	574	291	291
query97	2742	2793	2668	2668
query98	225	208	204	204
query99	1486	1425	1267	1267
Total cold run time: 279981 ms
Total hot run time: 186650 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5228b9d402770cb094f2c20736f4d5377001c3fb, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.61	0.11	0.11
query5	0.45	0.42	0.41
query6	1.16	0.65	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.60	0.52	0.50
query10	0.57	0.57	0.57
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.61
query14	0.78	0.81	0.80
query15	0.89	0.87	0.88
query16	0.37	0.38	0.40
query17	1.06	1.07	1.03
query18	0.23	0.21	0.20
query19	2.00	1.87	1.90
query20	0.02	0.01	0.01
query21	15.39	0.91	0.54
query22	0.75	1.13	0.61
query23	15.04	1.37	0.62
query24	6.67	1.16	0.83
query25	0.51	0.22	0.07
query26	0.75	0.16	0.13
query27	0.06	0.04	0.05
query28	9.53	0.88	0.43
query29	12.52	3.94	3.32
query30	0.25	0.09	0.07
query31	2.82	0.61	0.37
query32	3.24	0.55	0.46
query33	3.08	3.08	3.13
query34	15.96	5.38	4.74
query35	4.80	4.80	4.77
query36	0.70	0.51	0.48
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.66 s
Total hot run time: 29.35 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/21) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.14% (15397/26948)
Line Coverage 46.15% (139637/302542)
Region Coverage 45.47% (70761/155631)
Branch Coverage 40.23% (37342/92828)

@eldenmoon
Copy link
Member Author

run buildall

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33986 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5d70cc82d1a1a1db8c6cdf11c7ff3b515e73fe1, data reload: false

------ Round 1 ----------------------------------
q1	17619	5315	5072	5072
q2	1925	274	191	191
q3	10300	1276	708	708
q4	10214	1027	537	537
q5	7613	2458	2340	2340
q6	185	175	132	132
q7	899	760	602	602
q8	9325	1293	1137	1137
q9	6800	5079	5064	5064
q10	6958	2373	1983	1983
q11	479	287	279	279
q12	348	358	226	226
q13	17786	3689	3067	3067
q14	225	227	209	209
q15	563	484	471	471
q16	427	441	387	387
q17	630	858	373	373
q18	7500	7077	7096	7077
q19	1900	982	577	577
q20	334	332	231	231
q21	3706	3154	2374	2374
q22	1046	1003	949	949
Total cold run time: 106782 ms
Total hot run time: 33986 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5264	5211	5167	5167
q2	237	317	221	221
q3	2133	2660	2272	2272
q4	1379	1741	1338	1338
q5	4217	4524	4465	4465
q6	209	164	129	129
q7	2081	2009	1803	1803
q8	2628	2709	2622	2622
q9	7144	7251	7193	7193
q10	3163	3346	2917	2917
q11	591	516	499	499
q12	688	775	607	607
q13	3841	3927	3441	3441
q14	276	317	279	279
q15	512	487	472	472
q16	465	492	454	454
q17	1251	1583	1414	1414
q18	7818	7611	7776	7611
q19	854	828	881	828
q20	2077	1976	1829	1829
q21	4715	4442	4366	4366
q22	1062	1071	1009	1009
Total cold run time: 52605 ms
Total hot run time: 50936 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184873 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5d70cc82d1a1a1db8c6cdf11c7ff3b515e73fe1, data reload: false

query1	1013	391	397	391
query2	6532	1768	1801	1768
query3	6735	213	219	213
query4	26208	23373	23348	23348
query5	4352	594	422	422
query6	298	201	201	201
query7	4628	496	282	282
query8	270	218	214	214
query9	8593	2625	2618	2618
query10	451	365	280	280
query11	15344	15006	14801	14801
query12	164	105	104	104
query13	1651	532	403	403
query14	8617	5741	5771	5741
query15	207	198	169	169
query16	7244	637	460	460
query17	1222	743	587	587
query18	1977	397	309	309
query19	234	199	157	157
query20	118	123	113	113
query21	210	122	102	102
query22	4072	4144	4039	4039
query23	33880	32986	33001	32986
query24	8525	2350	2387	2350
query25	546	479	384	384
query26	1240	268	145	145
query27	2729	511	344	344
query28	4333	2141	2122	2122
query29	792	563	442	442
query30	284	221	193	193
query31	903	844	791	791
query32	71	68	64	64
query33	558	364	307	307
query34	796	833	531	531
query35	798	834	738	738
query36	946	1002	884	884
query37	111	99	77	77
query38	4145	4112	4118	4112
query39	1490	1425	1447	1425
query40	213	118	103	103
query41	56	55	51	51
query42	124	109	110	109
query43	511	512	475	475
query44	1323	843	822	822
query45	174	176	161	161
query46	828	1018	621	621
query47	1739	1789	1690	1690
query48	387	413	307	307
query49	723	492	389	389
query50	665	690	414	414
query51	4176	4137	4045	4045
query52	111	116	96	96
query53	228	245	192	192
query54	570	566	496	496
query55	82	90	82	82
query56	308	302	287	287
query57	1167	1178	1116	1116
query58	271	247	260	247
query59	2745	2643	2584	2584
query60	322	328	301	301
query61	124	124	124	124
query62	818	708	669	669
query63	236	193	195	193
query64	4401	995	690	690
query65	4239	4190	4144	4144
query66	1150	397	318	318
query67	15639	15354	15228	15228
query68	7970	887	532	532
query69	471	300	276	276
query70	1168	1103	1146	1103
query71	471	329	299	299
query72	5624	4803	4951	4803
query73	705	662	350	350
query74	9175	8907	8897	8897
query75	3799	3183	2678	2678
query76	3457	1132	701	701
query77	797	394	297	297
query78	9979	10210	9225	9225
query79	2024	831	590	590
query80	640	517	454	454
query81	485	255	233	233
query82	433	130	97	97
query83	255	245	242	242
query84	245	115	88	88
query85	812	346	312	312
query86	331	304	301	301
query87	4408	4457	4275	4275
query88	3678	2324	2305	2305
query89	391	319	284	284
query90	1923	213	214	213
query91	140	143	120	120
query92	78	58	56	56
query93	1698	961	584	584
query94	679	406	304	304
query95	383	295	280	280
query96	504	572	288	288
query97	2707	2763	2632	2632
query98	235	204	204	204
query99	1349	1424	1279	1279
Total cold run time: 272631 ms
Total hot run time: 184873 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5d70cc82d1a1a1db8c6cdf11c7ff3b515e73fe1, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.25	0.07	0.07
query4	1.63	0.11	0.12
query5	0.45	0.44	0.40
query6	1.17	0.67	0.67
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.60	0.50	0.54
query10	0.56	0.58	0.56
query11	0.15	0.12	0.11
query12	0.15	0.12	0.11
query13	0.65	0.61	0.61
query14	0.82	0.82	0.82
query15	0.92	0.88	0.88
query16	0.38	0.38	0.38
query17	1.06	1.05	1.06
query18	0.23	0.21	0.22
query19	1.98	1.86	1.81
query20	0.02	0.01	0.02
query21	15.38	0.92	0.55
query22	0.75	1.31	0.75
query23	14.74	1.39	0.66
query24	7.03	1.70	0.96
query25	0.47	0.12	0.22
query26	0.60	0.17	0.13
query27	0.08	0.06	0.05
query28	9.87	0.92	0.44
query29	12.53	4.00	3.40
query30	0.25	0.08	0.06
query31	2.85	0.60	0.38
query32	3.23	0.55	0.47
query33	3.13	3.11	3.12
query34	16.05	5.45	4.79
query35	4.81	4.87	4.81
query36	0.66	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.02	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.11 s
Total hot run time: 29.9 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.15% (15399/26947)
Line Coverage 46.17% (139647/302443)
Region Coverage 45.48% (70778/155617)
Branch Coverage 40.23% (37330/92792)

Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2025

PR approved by anyone and no changes requested.

Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2025
@eldenmoon eldenmoon merged commit 7d15c93 into apache:master Jul 16, 2025
24 of 25 checks passed
@eldenmoon eldenmoon deleted the simdjson-arch branch July 16, 2025 07:28
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 25, 2025
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 25, 2025
morrySnow pushed a commit that referenced this pull request Jul 28, 2025
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 30, 2025
dataroaring pushed a commit that referenced this pull request Aug 12, 2025
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.8-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants