Skip to content

[opt](load) avoid over-flushing active memtables in memory limiter (#52906)#53909

Merged
dataroaring merged 2 commits intoapache:branch-3.0from
kaijchen:pick-52906-3.0
Aug 13, 2025
Merged

[opt](load) avoid over-flushing active memtables in memory limiter (#52906)#53909
dataroaring merged 2 commits intoapache:branch-3.0from
kaijchen:pick-52906-3.0

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Jul 25, 2025 •
edited
Loading

backport #52906

@kaijchen
[opt](load) avoid over-flushing active memtables in memory limiter (a…
0b87962 
…pache#52906)

Related PR: apache#41018

Problem Summary:

Update the `_need_flush()` function to subtract both `_queue_mem_usage`
and `_flush_mem_usage` when deciding how much memory needs to be
flushed.

Previously, we only subtracted `_queue_mem_usage`, which could lead to
flushing more active memtables than necessary.

This change ensures that only the required amount of active memtable
memory is flushed, avoiding premature flushes.
This helps prevent creating small segments, which can hurt performance
and storage efficiency.
@kaijchen kaijchen requested a review from dataroaring as a code owner July 25, 2025 09:07
@Thearas
Copy link
Contributor

Thearas commented Jul 25, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

kaijchen commented Jul 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 25, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.90% (11136/26580)
Line Coverage 32.43% (95401/294132)
Region Coverage 31.56% (49258/156069)
Branch Coverage 27.99% (25245/90186)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 25, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.98% (19670/26234)
Line Coverage 68.24% (200447/293738)
Region Coverage 66.30% (119824/180737)
Branch Coverage 59.64% (60800/101952)

@kaijchen
Copy link
Member Author

kaijchen commented Jul 31, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 31, 2025

TPC-H: Total hot run time: 39415 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 54d129f6717d316cdeb48fa026731bf8b6d5c2c6, data reload: false

------ Round 1 ----------------------------------
q1	17900	6961	6616	6616
q2	2098	163	150	150
q3	10690	1121	1109	1109
q4	10495	735	755	735
q5	7782	2843	2790	2790
q6	221	133	133	133
q7	968	614	595	595
q8	9578	1953	1990	1953
q9	8092	6390	6390	6390
q10	7009	2195	2309	2195
q11	452	253	258	253
q12	399	206	205	205
q13	17809	2963	3000	2963
q14	250	201	210	201
q15	509	474	477	474
q16	510	375	374	374
q17	986	570	558	558
q18	7141	6653	6489	6489
q19	1398	1122	966	966
q20	485	197	199	197
q21	3791	3126	3089	3089
q22	1112	980	990	980
Total cold run time: 109675 ms
Total hot run time: 39415 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6674	6562	6662	6562
q2	320	238	231	231
q3	2889	2862	2925	2862
q4	2037	1820	1801	1801
q5	5660	5728	5715	5715
q6	215	129	128	128
q7	2234	1784	1830	1784
q8	3315	3539	3478	3478
q9	8757	8829	8812	8812
q10	3602	3530	3532	3530
q11	594	506	490	490
q12	821	591	601	591
q13	3926	3182	3118	3118
q14	307	263	287	263
q15	516	471	457	457
q16	504	462	437	437
q17	1856	1614	1593	1593
q18	8154	7742	7649	7649
q19	1643	1512	1531	1512
q20	2110	1875	1880	1875
q21	5205	5024	4940	4940
q22	1177	1016	1027	1016
Total cold run time: 62516 ms
Total hot run time: 58844 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2025

TPC-DS: Total hot run time: 196174 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 54d129f6717d316cdeb48fa026731bf8b6d5c2c6, data reload: false

query1	1274	922	904	904
query2	6253	1947	1904	1904
query3	10956	4461	4720	4461
query4	33444	23474	23387	23387
query5	3410	446	436	436
query6	263	168	170	168
query7	3981	318	315	315
query8	276	235	227	227
query9	9491	2552	2536	2536
query10	476	280	246	246
query11	17880	15568	15172	15172
query12	161	102	103	102
query13	1558	423	422	422
query14	9014	6637	6639	6637
query15	220	181	199	181
query16	7690	487	450	450
query17	1578	578	602	578
query18	2053	351	322	322
query19	213	168	171	168
query20	123	114	116	114
query21	211	105	107	105
query22	4637	4372	4382	4372
query23	35061	34562	34207	34207
query24	12781	2850	2837	2837
query25	697	418	430	418
query26	1824	170	168	168
query27	3118	360	351	351
query28	7579	2135	2167	2135
query29	1181	463	448	448
query30	269	163	162	162
query31	1048	825	854	825
query32	99	60	57	57
query33	788	310	311	310
query34	1033	546	512	512
query35	853	734	731	731
query36	1088	940	960	940
query37	278	66	67	66
query38	4090	4025	3982	3982
query39	1515	1451	1494	1451
query40	262	99	95	95
query41	49	47	46	46
query42	108	98	102	98
query43	542	467	482	467
query44	1232	839	801	801
query45	183	165	167	165
query46	1169	737	714	714
query47	2022	1916	1905	1905
query48	480	383	412	383
query49	1114	391	406	391
query50	826	415	419	415
query51	7478	7368	7272	7272
query52	102	90	89	89
query53	270	183	194	183
query54	1269	471	469	469
query55	85	78	81	78
query56	288	248	238	238
query57	1314	1210	1211	1210
query58	224	204	209	204
query59	3266	3041	2998	2998
query60	287	269	263	263
query61	111	108	121	108
query62	869	688	688	688
query63	222	190	183	183
query64	5037	670	635	635
query65	3399	3261	3259	3259
query66	1430	297	308	297
query67	16196	15747	15609	15609
query68	4959	582	560	560
query69	434	262	263	262
query70	1176	1080	1091	1080
query71	357	256	244	244
query72	6173	3962	4166	3962
query73	741	340	344	340
query74	10599	8930	8880	8880
query75	3360	2640	2639	2639
query76	3067	1069	1046	1046
query77	376	278	261	261
query78	10410	9563	9651	9563
query79	1631	588	595	588
query80	1216	424	430	424
query81	545	222	219	219
query82	907	84	86	84
query83	225	141	142	141
query84	239	74	77	74
query85	1357	316	283	283
query86	423	303	258	258
query87	4377	4252	4219	4219
query88	3719	2399	2342	2342
query89	406	290	292	290
query90	1952	183	183	183
query91	192	148	193	148
query92	61	50	50	50
query93	1806	551	537	537
query94	922	305	297	297
query95	349	274	249	249
query96	608	272	276	272
query97	3277	3123	3121	3121
query98	212	204	197	197
query99	1482	1291	1311	1291
Total cold run time: 305856 ms
Total hot run time: 196174 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2025

ClickBench: Total hot run time: 29.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 54d129f6717d316cdeb48fa026731bf8b6d5c2c6, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.61	0.10	0.10
query5	0.53	0.53	0.53
query6	1.14	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.56	0.58
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	0.77	0.79	0.80
query15	0.84	0.83	0.81
query16	0.37	0.37	0.37
query17	1.04	1.03	1.06
query18	0.23	0.21	0.21
query19	1.97	1.85	1.78
query20	0.01	0.02	0.01
query21	15.40	0.56	0.56
query22	3.32	1.83	1.48
query23	17.14	0.86	1.01
query24	2.61	0.92	1.82
query25	0.28	0.27	0.05
query26	0.37	0.13	0.13
query27	0.05	0.04	0.04
query28	10.16	0.47	0.48
query29	12.58	3.27	3.28
query30	0.25	0.06	0.06
query31	2.85	0.37	0.37
query32	3.26	0.46	0.46
query33	2.95	2.94	3.00
query34	16.99	4.54	4.48
query35	4.61	4.58	4.60
query36	0.67	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.86 s
Total hot run time: 29.8 s

@doris-robot
Copy link

doris-robot commented Jul 31, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.90% (11136/26580)
Line Coverage 32.43% (95407/294183)
Region Coverage 31.56% (49264/156114)
Branch Coverage 27.99% (25242/90186)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 31, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.01% (19679/26234)
Line Coverage 68.26% (200566/293818)
Region Coverage 66.37% (120016/180816)
Branch Coverage 59.72% (60895/101966)

dataroaring
dataroaring approved these changes Aug 13, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit bd45f01 into apache:branch-3.0 Aug 13, 2025
22 of 25 checks passed
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kaijchen @Thearas @doris-robot @hello-stephen @dataroaring