Skip to content

branch-3.1: [fix](variant)fix top-array data into variant column #54396#54867

Merged
morrySnow merged 3 commits intoapache:branch-3.1from
amorynan:pick_54396_to_doris_branch-3.1
Aug 19, 2025
Merged

branch-3.1: [fix](variant)fix top-array data into variant column #54396#54867
morrySnow merged 3 commits intoapache:branch-3.1from
amorynan:pick_54396_to_doris_branch-3.1

Conversation

@amorynan
Copy link
Contributor

@amorynan amorynan commented Aug 15, 2025

picked from #54396

@amorynan amorynan requested a review from morrySnow as a code owner August 15, 2025 08:23
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@amorynan
Copy link
Contributor Author

run buildall

1 similar comment
@amorynan
Copy link
Contributor Author

run buildall

@amorynan amorynan changed the title [fix](variant)fix top-array data into variant column (#54396) branch-3.1[fix](variant)fix top-array data into variant column (#54396) Aug 16, 2025
amory added 2 commits August 16, 2025 19:53
This pr mainly solves the problem when we insert top-level nested array
data into variant columnIssue Number:
like :
```
mysql> insert into sv1 values (1, '[{"a": 1, "c": 1.1}, {"b": "1"}]');
we maintain the association information between Nested in array , in this case , we will keep same offset at a,b,c ,

mysql> select * from sv1;
+------+----------------------------------------------+
| k    | v                                            |
+------+----------------------------------------------+
|    1 | {"a":[1,null],"b":[null,"1"],"c":[1.1,null]} |
+------+----------------------------------------------+
```
but in top array we just do flatten array not fill the null value to
maintain the association information between Nested in array
like:
```
mysql> insert into sv1 values (16, '[{"a": 1, "b": 1}, {"b": 2}, {"b": 3}]');
Query OK, 1 row affected (0.15 sec)
{'label':'label_4b3ede8b449a4d9a_b55fdeebe13b741c', 'status':'VISIBLE', 'txnId':'6031'}

mysql> select v from sv1 where k=16;
+-----------------------+
| v                     |
+-----------------------+
| {"a":[1],"b":[1,2,3]} |
+-----------------------+
1 row in set (0.19 sec)
```
@amorynan amorynan force-pushed the pick_54396_to_doris_branch-3.1 branch from d0160c9 to fe9dd57 Compare August 16, 2025 11:56
@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32753 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe9dd576b9010f2ff470c215a217f3ebbe1c8cc6, data reload: false

------ Round 1 ----------------------------------
q1	17590	5532	5530	5530
q2	2024	382	283	283
q3	11878	1251	763	763
q4	10568	866	462	462
q5	9559	2336	2142	2142
q6	184	163	135	135
q7	915	760	609	609
q8	9368	1459	1133	1133
q9	5233	4986	4913	4913
q10	6770	2262	1831	1831
q11	445	282	261	261
q12	332	359	210	210
q13	17759	3602	3027	3027
q14	225	224	208	208
q15	529	481	470	470
q16	418	423	365	365
q17	609	852	375	375
q18	6760	6425	6298	6298
q19	1213	949	562	562
q20	344	336	204	204
q21	2801	2211	1992	1992
q22	1047	995	980	980
Total cold run time: 106571 ms
Total hot run time: 32753 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5640	5529	5577	5529
q2	240	337	244	244
q3	2318	2692	2433	2433
q4	1424	1863	1389	1389
q5	4515	5070	4959	4959
q6	167	162	130	130
q7	2099	1996	1805	1805
q8	2648	2860	2713	2713
q9	7211	7240	7160	7160
q10	3078	3255	2772	2772
q11	586	527	499	499
q12	661	769	627	627
q13	3393	3816	3164	3164
q14	301	310	282	282
q15	524	484	478	478
q16	442	491	426	426
q17	1244	1748	1292	1292
q18	7551	7356	7248	7248
q19	791	1112	1079	1079
q20	2024	2036	1915	1915
q21	5378	5002	4486	4486
q22	1063	1067	1017	1017
Total cold run time: 53298 ms
Total hot run time: 51647 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192066 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe9dd576b9010f2ff470c215a217f3ebbe1c8cc6, data reload: false

query1	971	403	408	403
query2	6222	2047	1906	1906
query3	8689	195	195	195
query4	33988	23939	23511	23511
query5	3668	601	441	441
query6	283	191	176	176
query7	4202	493	313	313
query8	297	245	253	245
query9	9285	2618	2597	2597
query10	476	319	257	257
query11	18066	15552	15318	15318
query12	142	110	104	104
query13	1554	549	422	422
query14	9018	6614	7194	6614
query15	234	190	178	178
query16	8084	650	543	543
query17	1584	801	582	582
query18	2112	431	318	318
query19	219	184	177	177
query20	124	121	117	117
query21	213	137	117	117
query22	4582	4721	4361	4361
query23	35039	34521	34137	34137
query24	7750	2712	2751	2712
query25	539	503	428	428
query26	829	288	179	179
query27	2309	486	357	357
query28	5064	2221	2177	2177
query29	713	608	455	455
query30	242	190	161	161
query31	1091	934	847	847
query32	88	90	60	60
query33	504	397	313	313
query34	743	854	522	522
query35	766	814	755	755
query36	1011	1067	960	960
query37	116	90	65	65
query38	4032	3962	4044	3962
query39	1531	1522	1480	1480
query40	208	118	112	112
query41	54	51	49	49
query42	123	99	100	99
query43	504	517	503	503
query44	1331	820	817	817
query45	191	187	172	172
query46	921	1060	671	671
query47	1987	1934	1909	1909
query48	418	423	350	350
query49	747	491	411	411
query50	681	710	429	429
query51	7269	7209	7283	7209
query52	101	107	95	95
query53	234	261	193	193
query54	547	559	479	479
query55	77	80	81	80
query56	272	288	274	274
query57	1309	1250	1210	1210
query58	248	223	226	223
query59	3157	3338	3083	3083
query60	298	286	275	275
query61	112	116	118	116
query62	789	763	701	701
query63	241	202	195	195
query64	3644	1008	644	644
query65	3386	3267	3310	3267
query66	815	406	312	312
query67	16509	15801	15660	15660
query68	7711	825	547	547
query69	504	307	265	265
query70	1225	1091	1088	1088
query71	379	287	262	262
query72	5751	3738	3807	3738
query73	632	740	347	347
query74	10210	9353	9026	9026
query75	3169	3120	2670	2670
query76	2946	1156	775	775
query77	484	354	276	276
query78	10311	10323	9607	9607
query79	3231	905	602	602
query80	806	526	441	441
query81	489	258	221	221
query82	582	118	88	88
query83	163	165	146	146
query84	285	98	81	81
query85	783	381	302	302
query86	382	316	305	305
query87	4319	4329	4237	4237
query88	4935	2398	2383	2383
query89	401	336	287	287
query90	1740	192	189	189
query91	144	154	109	109
query92	70	56	53	53
query93	2283	876	546	546
query94	683	417	312	312
query95	347	288	271	271
query96	496	607	289	289
query97	3177	3311	3138	3138
query98	222	202	196	196
query99	1494	1419	1266	1266
Total cold run time: 292907 ms
Total hot run time: 192066 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fe9dd576b9010f2ff470c215a217f3ebbe1c8cc6, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.10
query5	0.53	0.53	0.51
query6	1.16	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.52	0.50
query10	0.57	0.55	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.62	0.60
query14	0.76	0.79	0.79
query15	0.85	0.83	0.83
query16	0.39	0.39	0.38
query17	0.98	1.05	1.02
query18	0.23	0.23	0.22
query19	1.97	1.87	1.85
query20	0.02	0.01	0.01
query21	15.38	0.89	0.55
query22	0.73	0.75	0.60
query23	15.23	1.44	0.54
query24	3.57	0.58	0.38
query25	0.31	0.05	0.06
query26	0.26	0.15	0.13
query27	0.05	0.04	0.04
query28	12.89	1.05	0.42
query29	12.54	3.88	3.22
query30	0.25	0.09	0.06
query31	2.84	0.60	0.39
query32	3.23	0.54	0.45
query33	3.01	3.04	3.05
query34	16.46	5.20	4.59
query35	4.57	4.56	4.54
query36	0.64	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.38 s
Total hot run time: 28.25 s

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32727 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe516a5400cb17e3cf8d15ca39cdaa7bc91c7a07, data reload: false

------ Round 1 ----------------------------------
q1	17598	5508	5455	5455
q2	2015	408	280	280
q3	11938	1246	755	755
q4	10558	855	450	450
q5	9449	2337	2121	2121
q6	191	160	134	134
q7	885	760	603	603
q8	9319	1383	1116	1116
q9	5182	4917	4891	4891
q10	6758	2273	1825	1825
q11	472	281	263	263
q12	334	354	214	214
q13	17764	3623	2992	2992
q14	239	225	214	214
q15	537	465	472	465
q16	415	424	381	381
q17	602	856	366	366
q18	6953	6456	6448	6448
q19	1220	948	538	538
q20	336	334	212	212
q21	2794	2212	2009	2009
q22	1063	1027	995	995
Total cold run time: 106622 ms
Total hot run time: 32727 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5499	5971	5487	5487
q2	237	326	231	231
q3	2246	2641	2347	2347
q4	1319	1761	1403	1403
q5	4340	4964	4933	4933
q6	165	159	127	127
q7	2073	1933	1798	1798
q8	2609	2847	2722	2722
q9	7221	7207	7190	7190
q10	3084	3237	2732	2732
q11	584	507	488	488
q12	674	774	640	640
q13	3352	3802	3170	3170
q14	293	300	271	271
q15	519	478	468	468
q16	445	486	435	435
q17	1231	1925	1254	1254
q18	7616	7435	7313	7313
q19	800	1127	1030	1030
q20	1977	2024	1907	1907
q21	5367	4887	4534	4534
q22	1082	1052	1011	1011
Total cold run time: 52733 ms
Total hot run time: 51491 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192591 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe516a5400cb17e3cf8d15ca39cdaa7bc91c7a07, data reload: false

query1	966	389	385	385
query2	6278	1969	1916	1916
query3	8687	201	195	195
query4	33369	23581	23415	23415
query5	3688	585	437	437
query6	291	195	176	176
query7	4193	485	325	325
query8	309	237	240	237
query9	9523	2593	2584	2584
query10	479	318	262	262
query11	18039	15361	15147	15147
query12	157	120	102	102
query13	1557	543	417	417
query14	9283	7201	7561	7201
query15	279	194	179	179
query16	8114	633	534	534
query17	1584	793	604	604
query18	2174	440	325	325
query19	220	194	157	157
query20	132	120	123	120
query21	208	130	107	107
query22	4725	4832	4581	4581
query23	35293	34451	34200	34200
query24	7421	2766	2771	2766
query25	571	482	431	431
query26	803	289	184	184
query27	2042	484	364	364
query28	5298	2208	2167	2167
query29	731	606	471	471
query30	246	188	163	163
query31	994	920	828	828
query32	81	64	60	60
query33	497	374	342	342
query34	748	835	521	521
query35	780	798	738	738
query36	1031	1074	961	961
query37	108	98	67	67
query38	3966	3998	3962	3962
query39	1710	1468	1495	1468
query40	219	119	107	107
query41	53	49	48	48
query42	118	102	103	102
query43	513	514	478	478
query44	1330	810	841	810
query45	185	184	169	169
query46	884	1073	680	680
query47	2011	2009	1929	1929
query48	406	432	347	347
query49	749	494	403	403
query50	663	726	438	438
query51	7335	7297	7053	7053
query52	101	108	97	97
query53	230	261	193	193
query54	544	546	488	488
query55	83	77	82	77
query56	271	266	265	265
query57	1336	1273	1221	1221
query58	234	213	225	213
query59	3108	3271	3174	3174
query60	303	280	262	262
query61	114	106	116	106
query62	797	765	726	726
query63	239	202	199	199
query64	3875	973	621	621
query65	3337	3266	3339	3266
query66	835	414	307	307
query67	16247	15783	15731	15731
query68	7543	812	533	533
query69	499	308	276	276
query70	1184	1084	1076	1076
query71	373	289	261	261
query72	5633	3676	3868	3676
query73	620	731	362	362
query74	10228	9161	8986	8986
query75	3206	3160	2624	2624
query76	2949	1115	774	774
query77	483	363	277	277
query78	10236	10315	9601	9601
query79	3306	876	594	594
query80	724	514	429	429
query81	479	259	220	220
query82	574	120	90	90
query83	158	158	142	142
query84	247	102	82	82
query85	770	395	302	302
query86	395	320	307	307
query87	4344	4385	4201	4201
query88	5217	2416	2373	2373
query89	404	326	300	300
query90	1720	190	186	186
query91	139	140	110	110
query92	64	54	51	51
query93	2098	866	534	534
query94	686	424	313	313
query95	340	281	272	272
query96	485	607	288	288
query97	3210	3281	3208	3208
query98	219	206	205	205
query99	1501	1403	1329	1329
Total cold run time: 292729 ms
Total hot run time: 192591 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fe516a5400cb17e3cf8d15ca39cdaa7bc91c7a07, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.04	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.11
query5	0.53	0.49	0.52
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.50	0.49
query10	0.54	0.55	0.55
query11	0.13	0.10	0.11
query12	0.13	0.11	0.11
query13	0.60	0.60	0.58
query14	0.78	0.79	0.79
query15	0.85	0.82	0.83
query16	0.37	0.40	0.38
query17	1.05	1.05	1.05
query18	0.24	0.22	0.23
query19	1.98	1.87	1.88
query20	0.01	0.01	0.01
query21	15.38	0.93	0.55
query22	0.73	0.77	0.70
query23	15.10	1.40	0.49
query24	3.35	1.41	0.34
query25	0.14	0.14	0.04
query26	0.27	0.15	0.14
query27	0.04	0.04	0.04
query28	12.28	1.05	0.43
query29	12.71	3.88	3.22
query30	0.26	0.10	0.07
query31	2.82	0.57	0.38
query32	3.22	0.53	0.46
query33	2.96	3.01	3.05
query34	16.68	5.09	4.59
query35	4.57	4.57	4.54
query36	0.63	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.14	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 102.57 s
Total hot run time: 28.24 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 80.46% (70/87) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.49% (12722/27967)
Line Coverage 36.37% (113409/311794)
Region Coverage 34.01% (64919/190877)
Branch Coverage 31.04% (34065/109754)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 82.76% (72/87) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.24% (21001/27545)
Line Coverage 69.60% (216651/311273)
Region Coverage 67.63% (129695/191777)
Branch Coverage 61.14% (67453/110318)

@morrySnow morrySnow changed the title branch-3.1[fix](variant)fix top-array data into variant column (#54396) branch-3.1: [fix](variant)fix top-array data into variant column #54396 Aug 19, 2025
@morrySnow morrySnow merged commit 394df08 into apache:branch-3.1 Aug 19, 2025
21 of 22 checks passed
@morrySnow morrySnow mentioned this pull request Aug 19, 2025
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

Comments