[opt](load) avoid over-flushing active memtables in memory limiter

[kaijchen]

What problem does this PR solve?

Related PR: #41018

Problem Summary:

Update the _need_flush() function to subtract both _queue_mem_usage and _flush_mem_usage when deciding how much memory needs to be flushed.

Previously, we only subtracted _queue_mem_usage, which could lead to flushing more active memtables than necessary.

This change ensures that only the required amount of active memtable memory is flushed, avoiding premature flushes.
This helps prevent creating small segments, which can hurt performance and storage efficiency.

Release note

None

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[Thearas]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[kaijchen]

run buildall

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[doris-robot]

TPC-H: Total hot run time: 33042 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca8caf6cc6e7de6ab42ac4e9d5d34b1550e57f61, data reload: false

------ Round 1 ----------------------------------
q1	17580	5254	5053	5053
q2	1935	265	178	178
q3	10346	1307	708	708
q4	10250	1004	507	507
q5	8108	2355	2307	2307
q6	181	160	127	127
q7	875	738	602	602
q8	9326	1279	1101	1101
q9	6813	5108	5147	5108
q10	6867	2375	1986	1986
q11	486	285	270	270
q12	336	342	214	214
q13	17764	3622	3065	3065
q14	222	228	214	214
q15	559	478	472	472
q16	417	417	382	382
q17	573	834	359	359
q18	7425	7091	7096	7091
q19	1943	958	542	542
q20	317	330	219	219
q21	3645	2533	2253	2253
q22	359	309	284	284
Total cold run time: 106327 ms
Total hot run time: 33042 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5094	5076	5066	5066
q2	249	320	218	218
q3	2160	2648	2275	2275
q4	1360	1750	1364	1364
q5	4165	4134	4443	4134
q6	220	174	124	124
q7	2020	1959	1872	1872
q8	2716	2508	2601	2508
q9	7130	7184	7246	7184
q10	3093	3311	2867	2867
q11	564	527	494	494
q12	695	1020	616	616
q13	3392	3802	3511	3511
q14	289	288	265	265
q15	512	486	468	468
q16	432	497	447	447
q17	1116	1503	1520	1503
q18	7959	7692	7537	7537
q19	780	768	844	768
q20	2010	2047	1919	1919
q21	4915	4348	4294	4294
q22	620	574	541	541
Total cold run time: 51491 ms
Total hot run time: 49975 ms

[doris-robot]

TPC-DS: Total hot run time: 184378 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca8caf6cc6e7de6ab42ac4e9d5d34b1550e57f61, data reload: false

query1	1003	402	384	384
query2	6550	1739	1745	1739
query3	6744	207	219	207
query4	26197	23562	22856	22856
query5	4346	593	453	453
query6	300	209	194	194
query7	4621	492	295	295
query8	290	219	208	208
query9	8584	2599	2624	2599
query10	474	320	260	260
query11	15935	15074	14779	14779
query12	156	106	105	105
query13	1652	540	401	401
query14	9029	5857	5868	5857
query15	205	194	169	169
query16	7305	423	267	267
query17	1327	684	565	565
query18	1974	384	285	285
query19	196	180	150	150
query20	117	114	113	113
query21	214	126	102	102
query22	4148	4018	3918	3918
query23	33741	33143	33026	33026
query24	8476	2352	2338	2338
query25	529	479	411	411
query26	1241	267	149	149
query27	2759	514	340	340
query28	4297	2118	2108	2108
query29	757	584	449	449
query30	284	222	198	198
query31	923	851	757	757
query32	74	67	65	65
query33	551	337	315	315
query34	785	844	514	514
query35	606	651	554	554
query36	954	963	877	877
query37	112	99	88	88
query38	4108	4072	4061	4061
query39	1674	1419	1427	1419
query40	216	121	107	107
query41	61	60	57	57
query42	136	113	111	111
query43	561	512	483	483
query44	1299	806	811	806
query45	171	171	162	162
query46	826	1008	626	626
query47	1731	1779	1711	1711
query48	381	411	307	307
query49	743	516	365	365
query50	623	694	413	413
query51	4168	4230	4132	4132
query52	111	107	97	97
query53	223	268	179	179
query54	572	572	495	495
query55	83	80	78	78
query56	307	293	286	286
query57	1175	1188	1112	1112
query58	255	252	243	243
query59	2581	2695	2580	2580
query60	318	323	304	304
query61	125	122	120	120
query62	805	738	672	672
query63	231	187	190	187
query64	4376	1148	844	844
query65	4259	4152	4167	4152
query66	1124	405	305	305
query67	15775	15363	15238	15238
query68	8485	922	523	523
query69	502	308	263	263
query70	1229	1122	1077	1077
query71	464	316	295	295
query72	5547	4628	4535	4535
query73	723	619	344	344
query74	8897	9006	8635	8635
query75	3779	3143	2665	2665
query76	3616	1142	744	744
query77	783	379	293	293
query78	10917	11124	10219	10219
query79	1175	873	602	602
query80	594	531	441	441
query81	453	270	217	217
query82	393	123	98	98
query83	256	247	238	238
query84	239	105	93	93
query85	828	349	310	310
query86	341	298	263	263
query87	4440	4455	4449	4449
query88	2840	2250	2250	2250
query89	385	310	277	277
query90	2025	222	203	203
query91	130	136	110	110
query92	76	63	54	54
query93	1058	953	595	595
query94	674	316	197	197
query95	371	287	279	279
query96	491	577	273	273
query97	2710	2793	2620	2620
query98	216	212	203	203
query99	1434	1410	1266	1266
Total cold run time: 272545 ms
Total hot run time: 184378 ms

[doris-robot]

ClickBench: Total hot run time: 29.52 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca8caf6cc6e7de6ab42ac4e9d5d34b1550e57f61, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.61	0.11	0.12
query5	0.43	0.41	0.43
query6	1.18	0.68	0.65
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.52	0.51
query10	0.57	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.61	0.61
query14	0.79	0.82	0.80
query15	0.90	0.87	0.86
query16	0.38	0.38	0.38
query17	1.06	1.05	1.05
query18	0.23	0.22	0.22
query19	1.93	1.84	1.90
query20	0.02	0.01	0.01
query21	15.39	0.89	0.56
query22	0.77	1.20	0.72
query23	14.80	1.34	0.62
query24	7.12	1.39	0.58
query25	0.50	0.24	0.15
query26	0.58	0.17	0.14
query27	0.06	0.05	0.06
query28	9.59	0.88	0.44
query29	12.55	3.92	3.34
query30	0.25	0.10	0.07
query31	2.83	0.60	0.38
query32	3.23	0.56	0.48
query33	3.08	3.06	3.12
query34	16.16	5.44	4.83
query35	4.91	4.91	4.86
query36	0.66	0.54	0.49
query37	0.09	0.07	0.07
query38	0.06	0.05	0.03
query39	0.03	0.03	0.03
query40	0.18	0.15	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.05 s
Total hot run time: 29.52 s

[doris-robot]

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.32% (15500/27039)
Line Coverage	46.30% (140875/304248)
Region Coverage	45.55% (71227/156355)
Branch Coverage	40.27% (37534/93200)

[dataroaring]

LGTM

[kaijchen]

run buildall

[doris-robot]

TPC-H: Total hot run time: 33236 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5adc2963f4ae4e789d44b4489165337b65a1062b, data reload: false

------ Round 1 ----------------------------------
q1	17623	5225	5117	5117
q2	1930	282	185	185
q3	10379	1274	709	709
q4	10298	1040	517	517
q5	8590	2393	2328	2328
q6	215	162	130	130
q7	910	752	607	607
q8	9312	1274	1115	1115
q9	6834	5115	5070	5070
q10	6920	2388	1935	1935
q11	486	316	281	281
q12	343	359	213	213
q13	17753	3742	3086	3086
q14	222	230	213	213
q15	554	489	482	482
q16	421	419	380	380
q17	624	846	393	393
q18	7412	7112	7186	7112
q19	1344	950	538	538
q20	334	360	223	223
q21	3735	3152	2306	2306
q22	354	326	296	296
Total cold run time: 106593 ms
Total hot run time: 33236 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5192	5121	5154	5121
q2	251	320	218	218
q3	2180	2654	2317	2317
q4	1391	1803	1332	1332
q5	4217	4370	4567	4370
q6	219	176	132	132
q7	2008	1963	1850	1850
q8	2601	2595	2591	2591
q9	7276	7209	7384	7209
q10	3105	3364	2926	2926
q11	631	535	491	491
q12	699	811	655	655
q13	3829	4102	3500	3500
q14	289	307	289	289
q15	515	486	482	482
q16	465	499	442	442
q17	1192	1622	1337	1337
q18	8180	7743	7576	7576
q19	832	808	896	808
q20	1943	1940	1837	1837
q21	4879	4409	4275	4275
q22	634	599	546	546
Total cold run time: 52528 ms
Total hot run time: 50304 ms

[doris-robot]

TPC-DS: Total hot run time: 186265 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5adc2963f4ae4e789d44b4489165337b65a1062b, data reload: false

query1	1000	396	386	386
query2	6537	1752	1789	1752
query3	6746	216	216	216
query4	26844	23715	22957	22957
query5	4328	571	438	438
query6	304	241	204	204
query7	4627	509	301	301
query8	286	229	211	211
query9	8641	2627	2642	2627
query10	479	326	286	286
query11	15756	15113	14876	14876
query12	161	109	104	104
query13	1654	542	397	397
query14	9233	5988	6019	5988
query15	205	193	175	175
query16	7678	440	263	263
query17	1307	715	582	582
query18	2032	417	316	316
query19	196	195	171	171
query20	124	123	114	114
query21	212	126	116	116
query22	4121	4185	4061	4061
query23	34099	33227	33237	33227
query24	8443	2402	2424	2402
query25	532	492	398	398
query26	1208	269	152	152
query27	2731	512	348	348
query28	4260	2136	2115	2115
query29	732	565	437	437
query30	310	224	192	192
query31	923	833	783	783
query32	76	67	59	59
query33	548	353	290	290
query34	807	846	531	531
query35	606	657	554	554
query36	947	979	873	873
query37	116	103	78	78
query38	4170	4069	4157	4069
query39	1473	1412	1434	1412
query40	217	119	109	109
query41	59	57	56	56
query42	134	119	105	105
query43	515	518	486	486
query44	1351	828	816	816
query45	176	173	165	165
query46	877	1037	624	624
query47	1727	1737	1711	1711
query48	372	412	314	314
query49	734	465	394	394
query50	649	675	412	412
query51	4178	4130	4102	4102
query52	108	106	101	101
query53	223	249	190	190
query54	575	568	499	499
query55	81	77	87	77
query56	297	302	303	302
query57	1181	1168	1106	1106
query58	271	265	259	259
query59	2636	2659	2676	2659
query60	325	311	312	311
query61	125	122	121	121
query62	835	684	673	673
query63	242	188	195	188
query64	4335	1197	812	812
query65	4324	4204	4187	4187
query66	1056	438	314	314
query67	15737	15537	15311	15311
query68	8099	887	528	528
query69	498	337	270	270
query70	1201	1118	1132	1118
query71	457	324	296	296
query72	5748	4769	5011	4769
query73	739	657	352	352
query74	9107	8843	9040	8843
query75	3735	3173	2718	2718
query76	3607	1157	725	725
query77	782	409	388	388
query78	11059	11036	10344	10344
query79	2015	833	592	592
query80	587	513	443	443
query81	479	254	218	218
query82	426	131	101	101
query83	259	249	233	233
query84	248	111	86	86
query85	787	361	308	308
query86	328	291	291	291
query87	4408	4434	4295	4295
query88	3388	2260	2259	2259
query89	378	323	291	291
query90	1928	218	214	214
query91	143	145	111	111
query92	74	63	58	58
query93	1242	947	592	592
query94	670	321	200	200
query95	373	289	287	287
query96	498	576	277	277
query97	2747	2774	2693	2693
query98	231	213	207	207
query99	1475	1408	1293	1293
Total cold run time: 275508 ms
Total hot run time: 186265 ms

[doris-robot]

ClickBench: Total hot run time: 30.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5adc2963f4ae4e789d44b4489165337b65a1062b, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.05	0.04
query3	0.26	0.08	0.07
query4	1.60	0.10	0.11
query5	0.45	0.42	0.43
query6	1.15	0.67	0.66
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.62	0.53	0.52
query10	0.57	0.57	0.58
query11	0.16	0.12	0.11
query12	0.16	0.12	0.12
query13	0.63	0.61	0.62
query14	0.81	0.82	0.84
query15	0.91	0.88	0.87
query16	0.39	0.40	0.39
query17	1.04	1.04	1.08
query18	0.23	0.21	0.20
query19	1.96	1.89	1.86
query20	0.02	0.01	0.01
query21	15.38	0.88	0.54
query22	0.76	1.21	0.66
query23	14.89	1.42	0.62
query24	6.80	1.35	1.18
query25	0.51	0.30	0.08
query26	0.55	0.16	0.13
query27	0.06	0.06	0.06
query28	10.73	0.94	0.45
query29	12.55	4.02	3.39
query30	0.25	0.09	0.06
query31	2.84	0.61	0.39
query32	3.24	0.56	0.48
query33	3.13	3.14	3.14
query34	16.01	5.46	4.82
query35	4.87	4.84	4.85
query36	0.67	0.51	0.49
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.91 s
Total hot run time: 30.14 s

[doris-robot]

BE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.46% (15518/27006)
Line Coverage	46.37% (141002/304111)
Region Coverage	45.63% (71336/156322)
Branch Coverage	40.34% (37596/93200)

[dataroaring]

run buildall

[doris-robot]

TPC-H: Total hot run time: 33293 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa80c13129b43c13ef4bfda25a4381681f67183e, data reload: false

------ Round 1 ----------------------------------
q1	17578	5264	5015	5015
q2	1939	282	201	201
q3	10420	1313	730	730
q4	10318	1019	551	551
q5	8740	2382	2309	2309
q6	208	164	131	131
q7	912	742	612	612
q8	9322	1300	1105	1105
q9	7404	5170	5205	5170
q10	6972	2370	1959	1959
q11	492	299	269	269
q12	369	354	217	217
q13	17787	3716	3076	3076
q14	220	235	214	214
q15	569	472	476	472
q16	432	414	385	385
q17	617	866	346	346
q18	7520	7099	7086	7086
q19	1436	956	536	536
q20	318	343	217	217
q21	3959	2549	2398	2398
q22	359	315	294	294
Total cold run time: 107891 ms
Total hot run time: 33293 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5209	5103	5119	5103
q2	243	318	226	226
q3	2175	2673	2279	2279
q4	1344	1775	1354	1354
q5	4247	4448	4524	4448
q6	215	173	132	132
q7	2053	1928	1857	1857
q8	2659	2573	2502	2502
q9	7363	7345	7270	7270
q10	3116	3318	2883	2883
q11	568	521	505	505
q12	721	783	637	637
q13	3590	4034	3296	3296
q14	298	303	304	303
q15	552	499	484	484
q16	460	494	469	469
q17	1196	1586	1420	1420
q18	8049	7655	7593	7593
q19	847	848	1041	848
q20	2083	2036	1948	1948
q21	4997	4632	4626	4626
q22	657	598	552	552
Total cold run time: 52642 ms
Total hot run time: 50735 ms

[doris-robot]

TPC-DS: Total hot run time: 187285 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa80c13129b43c13ef4bfda25a4381681f67183e, data reload: false

query1	983	392	401	392
query2	6504	1777	1804	1777
query3	6738	213	217	213
query4	26323	23519	23314	23314
query5	4344	577	429	429
query6	305	220	194	194
query7	4622	501	291	291
query8	254	220	220	220
query9	8618	2641	2627	2627
query10	461	320	262	262
query11	15477	15029	14855	14855
query12	162	108	105	105
query13	1662	535	411	411
query14	8528	5998	5848	5848
query15	209	188	175	175
query16	7445	447	272	272
query17	1311	730	600	600
query18	1989	419	315	315
query19	195	193	167	167
query20	124	121	117	117
query21	209	124	109	109
query22	4335	4496	4611	4496
query23	35175	33828	33843	33828
query24	8435	2397	2391	2391
query25	536	456	392	392
query26	1222	265	145	145
query27	2765	510	343	343
query28	4359	2128	2110	2110
query29	732	539	435	435
query30	285	210	187	187
query31	937	832	752	752
query32	67	58	59	58
query33	572	343	287	287
query34	816	851	522	522
query35	579	649	591	591
query36	928	972	878	878
query37	127	98	74	74
query38	4197	4070	4093	4070
query39	1454	1380	1399	1380
query40	203	114	103	103
query41	54	51	49	49
query42	129	108	107	107
query43	501	532	484	484
query44	1332	843	833	833
query45	173	171	169	169
query46	836	1054	637	637
query47	1758	1786	1733	1733
query48	380	424	326	326
query49	726	462	378	378
query50	630	709	409	409
query51	4215	4145	4214	4145
query52	116	111	95	95
query53	225	256	185	185
query54	584	571	506	506
query55	91	82	89	82
query56	309	301	280	280
query57	1199	1190	1087	1087
query58	267	260	256	256
query59	2746	2740	2626	2626
query60	334	323	308	308
query61	123	121	116	116
query62	777	718	649	649
query63	225	189	190	189
query64	4407	1265	851	851
query65	4247	4154	4180	4154
query66	1123	417	323	323
query67	15887	15705	15473	15473
query68	8833	910	533	533
query69	483	316	275	275
query70	1191	1122	1094	1094
query71	482	326	287	287
query72	5595	4819	4812	4812
query73	707	584	360	360
query74	8959	8828	8787	8787
query75	3968	3205	2655	2655
query76	3465	1158	740	740
query77	757	392	288	288
query78	10955	11077	10227	10227
query79	2050	819	594	594
query80	592	522	449	449
query81	480	268	222	222
query82	449	126	99	99
query83	246	246	244	244
query84	245	103	85	85
query85	780	408	317	317
query86	383	276	269	269
query87	4368	4454	4312	4312
query88	3367	2321	2309	2309
query89	387	322	277	277
query90	1839	211	212	211
query91	145	139	111	111
query92	74	58	58	58
query93	1617	962	598	598
query94	672	318	201	201
query95	383	299	282	282
query96	498	585	283	283
query97	2732	2751	2646	2646
query98	231	202	202	202
query99	1444	1501	1273	1273
Total cold run time: 275958 ms
Total hot run time: 187285 ms

[doris-robot]

ClickBench: Total hot run time: 29.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa80c13129b43c13ef4bfda25a4381681f67183e, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.08	0.07
query4	1.62	0.11	0.11
query5	0.44	0.42	0.42
query6	1.18	0.65	0.66
query7	0.02	0.02	0.01
query8	0.05	0.03	0.03
query9	0.61	0.53	0.53
query10	0.57	0.56	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.64	0.61	0.61
query14	0.80	0.82	0.81
query15	0.90	0.87	0.86
query16	0.38	0.39	0.39
query17	1.09	1.07	1.08
query18	0.22	0.21	0.22
query19	1.93	1.88	1.82
query20	0.01	0.01	0.02
query21	15.42	0.92	0.55
query22	0.77	1.06	0.83
query23	14.82	1.37	0.65
query24	6.99	2.40	0.43
query25	0.51	0.13	0.07
query26	0.71	0.17	0.14
query27	0.07	0.06	0.05
query28	9.00	0.89	0.43
query29	12.58	4.04	3.38
query30	0.28	0.09	0.07
query31	2.83	0.61	0.39
query32	3.25	0.57	0.48
query33	3.18	3.19	3.06
query34	16.14	5.40	4.80
query35	4.79	4.90	4.88
query36	0.70	0.51	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.65 s
Total hot run time: 29.49 s

[hello-stephen]

BE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.01% (15500/27189)
Line Coverage	45.86% (138183/301337)
Region Coverage	35.31% (103352/292670)
Branch Coverage	37.77% (45465/120377)

[liaoxin01]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[dataroaring]

LGTM