Skip to content

branch-3.0: [fix](iceberg) fix insert iceberg transform partition table problems and add some cases (#54557)#54796

Merged
dataroaring merged 2 commits intoapache:branch-3.0from
suxiaogang223:paimon_time_zone_3.0
Aug 16, 2025
Merged

branch-3.0: [fix](iceberg) fix insert iceberg transform partition table problems and add some cases (#54557)#54796
dataroaring merged 2 commits intoapache:branch-3.0from
suxiaogang223:paimon_time_zone_3.0

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Aug 14, 2025

bp: #54557

suxiaogang223 and others added 2 commits August 14, 2025 18:55
@suxiaogang223
[fix](iceberg) fix insert iceberg transform partition table problems …
2521bd5 
…and add some cases (apache#54557)

### What problem does this PR solve?

Problem Summary:
1. Fixed the problem with inserting into Iceberg decimal bucket
partition table due to memory issues
2. Fixed the exception caused by division by zero when selecting
count(*) from an empty iceberg partition table
3. Add cases to cover the logic of Iceberg transform partition table
https://iceberg.apache.org/spec/#partition-transforms
@suxiaogang223
remove some cases
3421a96
@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Aug 14, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-H: Total hot run time: 39701 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3421a96a9650bae5a55845ca0dca5592a3fdddca, data reload: false

------ Round 1 ----------------------------------
q1	17694	6847	6644	6644
q2	2042	222	199	199
q3	10581	1142	1207	1142
q4	10512	736	678	678
q5	7729	2879	2850	2850
q6	214	128	130	128
q7	969	616	611	611
q8	9338	1897	2009	1897
q9	7570	6353	6378	6353
q10	7052	2256	2274	2256
q11	461	269	259	259
q12	391	213	207	207
q13	17809	2984	2996	2984
q14	240	212	208	208
q15	516	464	470	464
q16	500	383	369	369
q17	981	547	522	522
q18	7276	6593	6577	6577
q19	1397	1093	1079	1079
q20	479	201	203	201
q21	4234	3060	3105	3060
q22	1070	1020	1013	1013
Total cold run time: 109055 ms
Total hot run time: 39701 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6619	6548	6569	6548
q2	334	238	235	235
q3	2900	2973	2909	2909
q4	2038	1838	1766	1766
q5	5702	5729	5709	5709
q6	209	124	130	124
q7	2221	1771	1808	1771
q8	3355	3552	3481	3481
q9	8713	8860	8810	8810
q10	3561	3469	3494	3469
q11	615	483	485	483
q12	791	579	576	576
q13	8302	3154	3117	3117
q14	302	266	261	261
q15	508	460	469	460
q16	481	442	432	432
q17	1838	1610	1608	1608
q18	8095	7768	7825	7768
q19	1653	1545	1522	1522
q20	2032	1835	1872	1835
q21	5145	5022	4972	4972
q22	1161	1088	997	997
Total cold run time: 66575 ms
Total hot run time: 58853 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-DS: Total hot run time: 192471 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3421a96a9650bae5a55845ca0dca5592a3fdddca, data reload: false

query1	951	380	377	377
query2	6221	1993	1910	1910
query3	8684	202	197	197
query4	33481	23854	23615	23615
query5	3603	481	461	461
query6	278	173	176	173
query7	4204	309	309	309
query8	294	220	228	220
query9	9457	2598	2614	2598
query10	488	262	262	262
query11	17940	15130	15361	15130
query12	163	110	108	108
query13	1574	449	440	440
query14	9739	7256	6675	6675
query15	255	175	187	175
query16	8069	518	499	499
query17	1637	606	613	606
query18	2199	339	317	317
query19	282	174	168	168
query20	125	124	118	118
query21	205	109	107	107
query22	4620	4519	4429	4429
query23	34536	34221	33927	33927
query24	11816	2923	2888	2888
query25	663	440	425	425
query26	1413	174	182	174
query27	3044	361	365	361
query28	7575	2131	2145	2131
query29	954	473	457	457
query30	266	159	161	159
query31	1070	827	850	827
query32	98	59	61	59
query33	763	343	310	310
query34	988	508	528	508
query35	850	749	744	744
query36	1150	947	977	947
query37	129	69	70	69
query38	4104	4031	3992	3992
query39	1538	1474	1493	1474
query40	254	101	105	101
query41	49	48	49	48
query42	112	99	106	99
query43	528	501	495	495
query44	1289	824	833	824
query45	188	166	177	166
query46	1159	724	747	724
query47	2005	1877	1913	1877
query48	507	389	395	389
query49	909	414	405	405
query50	856	433	434	433
query51	7384	7422	7292	7292
query52	101	90	95	90
query53	271	188	186	186
query54	1169	477	501	477
query55	82	89	84	84
query56	295	259	252	252
query57	1311	1227	1212	1212
query58	226	207	203	203
query59	3276	3098	3090	3090
query60	286	254	255	254
query61	112	153	111	111
query62	864	687	731	687
query63	238	195	202	195
query64	5188	686	650	650
query65	3447	3272	3214	3214
query66	1302	301	304	301
query67	15804	15718	15451	15451
query68	4970	578	599	578
query69	443	264	272	264
query70	1200	1090	1119	1090
query71	383	256	276	256
query72	6232	4124	4125	4124
query73	749	368	355	355
query74	10286	9377	9260	9260
query75	3379	2662	2662	2662
query76	2818	1009	1077	1009
query77	397	284	267	267
query78	10461	9665	9513	9513
query79	2411	612	623	612
query80	959	416	420	416
query81	549	224	222	222
query82	1007	91	86	86
query83	244	141	141	141
query84	236	81	81	81
query85	1598	308	292	292
query86	476	306	294	294
query87	4390	4317	4214	4214
query88	4401	2382	2359	2359
query89	428	298	298	298
query90	1974	189	189	189
query91	188	148	150	148
query92	61	50	50	50
query93	2481	574	562	562
query94	853	294	298	294
query95	370	265	268	265
query96	627	289	283	283
query97	3312	3145	3182	3145
query98	214	201	200	200
query99	1555	1323	1286	1286
Total cold run time: 304290 ms
Total hot run time: 192471 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

ClickBench: Total hot run time: 30.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3421a96a9650bae5a55845ca0dca5592a3fdddca, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.67	0.10	0.11
query5	0.51	0.51	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.51	0.52
query10	0.56	0.56	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.11
query13	0.62	0.61	0.59
query14	0.78	0.79	0.80
query15	0.85	0.83	0.83
query16	0.38	0.38	0.38
query17	1.05	1.05	1.06
query18	0.25	0.22	0.22
query19	1.96	1.87	1.79
query20	0.01	0.01	0.01
query21	15.38	0.57	0.58
query22	2.42	2.50	1.78
query23	16.72	0.98	0.84
query24	3.03	1.64	1.42
query25	0.21	0.10	0.04
query26	0.55	0.14	0.13
query27	0.04	0.04	0.04
query28	9.52	0.50	0.50
query29	12.57	3.24	3.24
query30	0.24	0.06	0.06
query31	2.86	0.39	0.38
query32	3.24	0.46	0.46
query33	3.00	3.04	3.07
query34	16.92	4.59	4.51
query35	4.56	4.56	4.51
query36	0.67	0.47	0.50
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.42 s
Total hot run time: 30.72 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 42.06% (11232/26705)
Line Coverage 32.56% (96076/295048)
Region Coverage 30.50% (55159/180838)
Branch Coverage 26.82% (27286/101744)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.10% (19712/26246)
Line Coverage 68.35% (201137/294264)
Region Coverage 66.50% (120443/181106)
Branch Coverage 59.82% (61097/102134)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 15, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.10% (19712/26246)
Line Coverage 68.35% (201137/294264)
Region Coverage 66.50% (120443/181106)
Branch Coverage 59.82% (61097/102134)

dataroaring
dataroaring approved these changes Aug 16, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit c865166 into apache:branch-3.0 Aug 16, 2025
22 of 24 checks passed
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
@suxiaogang223 suxiaogang223 deleted the paimon_time_zone_3.0 branch September 23, 2025 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@suxiaogang223 @hello-stephen @doris-robot @dataroaring

Comments