Skip to content

branch-3.1: [fix](iceberg) fix insert iceberg transform partition table problems and add some cases #54557#54723

Merged
morrySnow merged 1 commit intobranch-3.1from
auto-pick-54557-branch-3.1
Aug 15, 2025
Merged

branch-3.1: [fix](iceberg) fix insert iceberg transform partition table problems and add some cases #54557#54723
morrySnow merged 1 commit intobranch-3.1from
auto-pick-54557-branch-3.1

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 13, 2025

Cherry-picked from #54557

@suxiaogang223
[fix](iceberg) fix insert iceberg transform partition table problems …
562b28a 
…and add some cases (#54557)

### What problem does this PR solve?

Problem Summary:
1. Fixed the problem with inserting into Iceberg decimal bucket
partition table due to memory issues
2. Fixed the exception caused by division by zero when selecting
count(*) from an empty iceberg partition table
3. Add cases to cover the logic of Iceberg transform partition table
https://iceberg.apache.org/spec/#partition-transforms
@github-actions github-actions bot requested a review from morrySnow as a code owner August 13, 2025 23:53
@morningman
Copy link
Contributor

morningman commented Aug 14, 2025

run

@morningman morningman closed this Aug 14, 2025
@morningman morningman reopened this Aug 14, 2025
@morningman
Copy link
Contributor

morningman commented Aug 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-H: Total hot run time: 32499 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 562b28a01b39afed20fb36a8deb67a978bf125d1, data reload: false

------ Round 1 ----------------------------------
q1	17586	5527	5448	5448
q2	2043	311	173	173
q3	10401	1307	716	716
q4	10209	872	443	443
q5	7609	2419	2143	2143
q6	182	170	134	134
q7	878	757	628	628
q8	9340	1491	1190	1190
q9	5275	4946	4895	4895
q10	6766	2267	1809	1809
q11	476	292	272	272
q12	330	352	208	208
q13	17772	3600	3045	3045
q14	229	229	209	209
q15	529	476	457	457
q16	417	425	382	382
q17	607	871	368	368
q18	7002	6316	6460	6316
q19	1197	960	537	537
q20	327	352	211	211
q21	2889	2182	1916	1916
q22	1062	1050	999	999
Total cold run time: 103126 ms
Total hot run time: 32499 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5522	5460	5462	5460
q2	236	325	228	228
q3	2290	2664	2307	2307
q4	1331	1777	1306	1306
q5	4394	4838	5002	4838
q6	174	164	126	126
q7	2085	2013	1816	1816
q8	2605	2806	2711	2711
q9	7197	7127	7172	7127
q10	3048	3280	2799	2799
q11	585	506	491	491
q12	678	731	598	598
q13	3338	3791	3170	3170
q14	271	297	266	266
q15	526	470	456	456
q16	429	481	425	425
q17	1220	1763	1278	1278
q18	7662	7532	7247	7247
q19	787	982	1140	982
q20	1994	2040	1871	1871
q21	5311	4920	4606	4606
q22	1105	1124	1021	1021
Total cold run time: 52788 ms
Total hot run time: 51129 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-DS: Total hot run time: 190746 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 562b28a01b39afed20fb36a8deb67a978bf125d1, data reload: false

query1	966	382	384	382
query2	6537	1958	1956	1956
query3	6709	210	215	210
query4	33902	24218	23474	23474
query5	4342	619	449	449
query6	277	190	180	180
query7	4636	520	325	325
query8	301	255	234	234
query9	9571	2590	2582	2582
query10	490	338	266	266
query11	18004	15373	15585	15373
query12	154	107	107	107
query13	1658	536	415	415
query14	10118	7047	7014	7014
query15	258	191	178	178
query16	8027	667	466	466
query17	1661	786	586	586
query18	2094	415	322	322
query19	239	189	167	167
query20	123	120	119	119
query21	209	129	110	110
query22	4660	4774	4671	4671
query23	34430	33451	33253	33253
query24	7314	2662	2606	2606
query25	496	488	405	405
query26	1216	277	170	170
query27	1994	470	334	334
query28	5022	2155	2115	2115
query29	712	586	456	456
query30	261	187	159	159
query31	920	882	789	789
query32	73	62	63	62
query33	509	364	309	309
query34	733	834	517	517
query35	761	800	723	723
query36	995	1051	936	936
query37	103	95	68	68
query38	3895	3888	3787	3787
query39	1471	1597	1432	1432
query40	224	121	104	104
query41	55	60	59	59
query42	118	110	107	107
query43	514	514	478	478
query44	1332	795	795	795
query45	183	179	174	174
query46	872	1046	660	660
query47	1920	1949	1913	1913
query48	474	416	327	327
query49	797	482	392	392
query50	673	695	424	424
query51	7331	7121	7150	7121
query52	99	101	95	95
query53	226	249	180	180
query54	529	531	464	464
query55	82	77	77	77
query56	273	273	254	254
query57	1229	1249	1191	1191
query58	247	222	215	215
query59	3049	3144	2900	2900
query60	284	285	274	274
query61	116	165	126	126
query62	782	725	673	673
query63	222	195	189	189
query64	4187	960	615	615
query65	3300	3196	3228	3196
query66	1078	409	312	312
query67	16102	15823	15762	15762
query68	7354	833	551	551
query69	470	309	270	270
query70	1150	1093	1113	1093
query71	536	291	263	263
query72	5599	3777	3716	3716
query73	634	754	349	349
query74	9773	9380	8848	8848
query75	3220	3141	2677	2677
query76	3272	1141	751	751
query77	502	364	282	282
query78	10318	10379	9657	9657
query79	3362	889	607	607
query80	694	526	459	459
query81	517	261	216	216
query82	644	120	91	91
query83	166	171	141	141
query84	236	102	82	82
query85	784	355	297	297
query86	409	317	303	303
query87	4306	4310	4246	4246
query88	5105	2403	2377	2377
query89	411	333	300	300
query90	1883	185	187	185
query91	136	142	108	108
query92	70	56	52	52
query93	1973	892	543	543
query94	698	416	297	297
query95	333	272	258	258
query96	501	627	289	289
query97	3190	3289	3161	3161
query98	227	207	197	197
query99	1529	1457	1296	1296
Total cold run time: 292141 ms
Total hot run time: 190746 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

ClickBench: Total hot run time: 28.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 562b28a01b39afed20fb36a8deb67a978bf125d1, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.53	0.52	0.51
query6	1.13	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.51
query10	0.56	0.56	0.54
query11	0.14	0.11	0.10
query12	0.13	0.11	0.11
query13	0.62	0.60	0.59
query14	0.77	0.80	0.79
query15	0.86	0.84	0.81
query16	0.41	0.39	0.39
query17	1.06	1.02	1.07
query18	0.24	0.22	0.22
query19	1.89	1.87	1.78
query20	0.01	0.02	0.01
query21	15.40	0.90	0.57
query22	0.73	0.83	0.57
query23	15.19	1.47	0.57
query24	3.39	1.23	0.76
query25	0.18	0.16	0.06
query26	0.26	0.14	0.13
query27	0.04	0.06	0.05
query28	13.70	1.06	0.43
query29	12.56	3.92	3.27
query30	0.25	0.10	0.06
query31	2.83	0.60	0.38
query32	3.22	0.53	0.46
query33	3.02	3.03	3.08
query34	16.56	5.10	4.59
query35	4.58	4.59	4.59
query36	0.64	0.50	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.95 s
Total hot run time: 28.66 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.44% (12707/27964)
Line Coverage 36.30% (113154/311677)
Region Coverage 33.95% (64783/190801)
Branch Coverage 30.99% (33994/109692)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.16% (20975/27542)
Line Coverage 69.54% (216388/311151)
Region Coverage 67.57% (129525/191696)
Branch Coverage 61.15% (67421/110250)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.40% (21317/27542)
Line Coverage 71.65% (222930/311151)
Region Coverage 69.62% (133453/191696)
Branch Coverage 63.23% (69715/110250)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.06% (20949/27542)
Line Coverage 69.43% (216046/311151)
Region Coverage 67.47% (129332/191696)
Branch Coverage 61.04% (67302/110250)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.41% (21319/27542)
Line Coverage 71.66% (222976/311151)
Region Coverage 69.65% (133513/191696)
Branch Coverage 63.25% (69738/110250)

@morrySnow morrySnow merged commit b210d1d into branch-3.1 Aug 15, 2025
22 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-54557-branch-3.1 branch August 15, 2025 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @doris-robot @hello-stephen @morrySnow @suxiaogang223