Skip to content

branch-3.1: [feat](compaction) Make base compaction output rowset write filecache more adaptive #54694#56278

Merged
morrySnow merged 2 commits intoapache:branch-3.1from
swjtu-zhanglei:branch-3.1-pick
Sep 25, 2025
Merged

branch-3.1: [feat](compaction) Make base compaction output rowset write filecache more adaptive #54694#56278
morrySnow merged 2 commits intoapache:branch-3.1from
swjtu-zhanglei:branch-3.1-pick

Conversation

@swjtu-zhanglei
Copy link
Contributor

@swjtu-zhanglei swjtu-zhanglei commented Sep 22, 2025 •
edited by morrySnow
Loading

Cherry-picked from #54694

@Thearas
Copy link
Contributor

Thearas commented Sep 22, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@swjtu-zhanglei
branch-3.1: [feat](compaction) Make base compaction output rowset wri…
879a268 
…te filecache more adaptive (apache#54694)

Cherry-picked from apache#54694

* when base compaction input rowsets (total_cache_size /
total_data_size) > file_cache_keep_base_compaction_output_min_hit_ratio
      output rowsets will write to filecache
@swjtu-zhanglei
branch-3.1: [fix](test) try to fix two filecache regresion test case (a…
652b3ce 
…pache#56195)

Cherry-picked from apache#56195

* make `test_filecache_with_base_compaction_thresthold` run faster and
use less memory
* fix test_list_cache_file not stable
@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Sep 22, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 22, 2025

TPC-H: Total hot run time: 32777 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 652b3ce091029e44dcb38784c8e4f7923d9109b1, data reload: false

------ Round 1 ----------------------------------
q1	17574	5529	5373	5373
q2	2036	398	297	297
q3	12120	1240	765	765
q4	10249	885	456	456
q5	8408	2378	2160	2160
q6	185	167	134	134
q7	903	753	606	606
q8	9339	1495	1196	1196
q9	5230	5010	4940	4940
q10	6782	2294	1820	1820
q11	487	286	273	273
q12	343	353	216	216
q13	17773	3603	3002	3002
q14	228	225	209	209
q15	526	462	458	458
q16	406	431	385	385
q17	592	870	372	372
q18	7159	6528	6392	6392
q19	1611	961	541	541
q20	337	331	208	208
q21	2812	2154	1994	1994
q22	1037	1010	980	980
Total cold run time: 106137 ms
Total hot run time: 32777 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5541	5495	5473	5473
q2	242	334	238	238
q3	2272	2630	2328	2328
q4	1337	1767	1381	1381
q5	4460	4979	4992	4979
q6	177	162	134	134
q7	2073	1994	1847	1847
q8	2657	2836	2718	2718
q9	7298	7209	7206	7206
q10	3037	3297	2675	2675
q11	555	495	497	495
q12	695	769	618	618
q13	3416	3795	3176	3176
q14	267	292	268	268
q15	518	474	470	470
q16	428	480	429	429
q17	1264	1733	1258	1258
q18	7712	7593	7470	7470
q19	826	1129	1104	1104
q20	2070	2053	1893	1893
q21	5286	4952	4692	4692
q22	1069	1028	958	958
Total cold run time: 53200 ms
Total hot run time: 51810 ms

@doris-robot
Copy link

doris-robot commented Sep 22, 2025

TPC-DS: Total hot run time: 192903 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 652b3ce091029e44dcb38784c8e4f7923d9109b1, data reload: false

query1	963	382	377	377
query2	6159	1972	1932	1932
query3	8688	204	196	196
query4	33544	23819	23749	23749
query5	4315	609	456	456
query6	306	205	178	178
query7	4208	504	333	333
query8	309	251	248	248
query9	9349	2643	2630	2630
query10	492	334	257	257
query11	18196	15437	15493	15437
query12	154	110	98	98
query13	1554	539	424	424
query14	9834	7200	6585	6585
query15	247	189	181	181
query16	8071	662	492	492
query17	1597	767	591	591
query18	2223	409	330	330
query19	225	194	162	162
query20	121	129	126	126
query21	205	130	113	113
query22	4479	4538	4469	4469
query23	35080	34633	34264	34264
query24	7493	2720	2693	2693
query25	530	504	418	418
query26	845	287	169	169
query27	2146	484	357	357
query28	5091	2258	2237	2237
query29	707	621	463	463
query30	245	194	159	159
query31	1034	933	843	843
query32	96	59	56	56
query33	490	353	308	308
query34	763	872	533	533
query35	792	791	759	759
query36	1023	1060	937	937
query37	121	103	66	66
query38	4074	3992	4004	3992
query39	1499	1452	1471	1452
query40	210	117	100	100
query41	49	45	47	45
query42	124	103	108	103
query43	520	526	498	498
query44	1318	864	837	837
query45	192	185	176	176
query46	878	1070	677	677
query47	2005	1979	1920	1920
query48	405	435	349	349
query49	753	517	421	421
query50	672	694	456	456
query51	7325	7437	7235	7235
query52	104	101	95	95
query53	239	265	198	198
query54	554	557	472	472
query55	79	80	82	80
query56	281	280	266	266
query57	1306	1258	1198	1198
query58	238	226	226	226
query59	3165	3357	3112	3112
query60	283	313	258	258
query61	116	114	111	111
query62	783	769	716	716
query63	233	192	188	188
query64	3762	1017	652	652
query65	3364	3320	3329	3320
query66	774	406	332	332
query67	16611	15861	15534	15534
query68	7863	827	554	554
query69	498	298	272	272
query70	1146	1142	1097	1097
query71	430	296	261	261
query72	5201	3748	3827	3748
query73	652	749	351	351
query74	10605	9206	9308	9206
query75	3807	3153	2654	2654
query76	3515	1181	782	782
query77	793	352	281	281
query78	10199	10435	9545	9545
query79	2704	900	611	611
query80	688	518	454	454
query81	490	256	218	218
query82	235	125	87	87
query83	173	161	142	142
query84	276	102	89	89
query85	748	377	318	318
query86	347	292	306	292
query87	4301	4300	4192	4192
query88	3540	2428	2409	2409
query89	422	324	296	296
query90	2028	187	191	187
query91	140	141	110	110
query92	69	55	55	55
query93	2004	866	547	547
query94	649	407	276	276
query95	342	284	273	273
query96	493	598	290	290
query97	3188	3356	3135	3135
query98	214	218	204	204
query99	1475	1425	1322	1322
Total cold run time: 292846 ms
Total hot run time: 192903 ms

@doris-robot
Copy link

doris-robot commented Sep 22, 2025

ClickBench: Total hot run time: 28.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 652b3ce091029e44dcb38784c8e4f7923d9109b1, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.53	0.52	0.49
query6	1.13	0.73	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	0.77	0.80	0.79
query15	0.85	0.82	0.83
query16	0.39	0.38	0.38
query17	1.08	0.98	1.00
query18	0.24	0.22	0.22
query19	1.96	1.76	1.79
query20	0.02	0.01	0.01
query21	15.44	0.96	0.55
query22	0.74	0.76	0.63
query23	15.18	1.52	0.54
query24	3.35	0.56	0.77
query25	0.11	0.04	0.14
query26	0.29	0.16	0.14
query27	0.05	0.05	0.04
query28	13.29	1.08	0.44
query29	12.58	3.90	3.27
query30	0.24	0.10	0.07
query31	2.82	0.63	0.38
query32	3.23	0.54	0.46
query33	3.01	3.03	3.02
query34	16.57	5.23	4.53
query35	4.57	4.59	4.58
query36	0.65	0.53	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.61 s
Total hot run time: 28.35 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 22, 2025

BE UT Coverage Report

Increment line coverage 32.14% (27/84) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.58% (12787/28057)
Line Coverage 36.40% (114056/313327)
Region Coverage 34.01% (65197/191679)
Branch Coverage 31.04% (34213/110218)

@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Sep 22, 2025

run p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 22, 2025

BE Regression && UT Coverage Report

Increment line coverage 92.86% (78/84) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.47% (21102/27594)
Line Coverage 69.80% (218023/312361)
Region Coverage 67.74% (130362/192456)
Branch Coverage 61.27% (67843/110732)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 22, 2025

BE Regression && UT Coverage Report

Increment line coverage 92.86% (78/84) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.47% (21102/27594)
Line Coverage 69.80% (218017/312361)
Region Coverage 67.73% (130359/192456)
Branch Coverage 61.27% (67848/110732)

@morrySnow morrySnow changed the title branch-3.1: [feat](compaction) Make base compaction output rowset write filecache more adaptive (#54694) branch-3.1: [feat](compaction) Make base compaction output rowset write filecache more adaptive #54694 Sep 25, 2025
@morrySnow morrySnow merged commit 57bfe70 into apache:branch-3.1 Sep 25, 2025
21 of 22 checks passed
@swjtu-zhanglei swjtu-zhanglei mentioned this pull request Sep 25, 2025
Merged
16 tasks
@morrySnow morrySnow mentioned this pull request Oct 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@swjtu-zhanglei @Thearas @doris-robot @hello-stephen @morrySnow