Skip to content

[improve](load) do not block delta writer if memtable memory is low (#42649)#54517

Merged
dataroaring merged 1 commit intoapache:branch-3.0from
kaijchen:pick-42649-3.0
Aug 12, 2025
Merged

[improve](load) do not block delta writer if memtable memory is low (#42649)#54517
dataroaring merged 1 commit intoapache:branch-3.0from
kaijchen:pick-42649-3.0

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Aug 9, 2025

cherry-pick #42649

…pache#42649)

## Proposed changes

Currently when the process memory usage is high, even if load memory
usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in `INSERT INTO SELECT`
statements.

This PR changes this behavior. If the total load memory usage is lower
than 5% (current default),
the load will no longer be blocked by hard limit.
@kaijchen kaijchen requested a review from dataroaring as a code owner August 9, 2025 07:31
@Thearas
Copy link
Contributor

Thearas commented Aug 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

kaijchen commented Aug 9, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39867 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b80612afeeaced57b77fbac9d63421f2f838554, data reload: false

------ Round 1 ----------------------------------
q1	17600	6881	6605	6605
q2	2093	161	170	161
q3	10616	1139	1136	1136
q4	10236	749	720	720
q5	7765	2840	2782	2782
q6	210	130	129	129
q7	942	635	599	599
q8	9353	1963	2014	1963
q9	6653	6407	6397	6397
q10	7025	2235	2257	2235
q11	460	268	257	257
q12	419	212	204	204
q13	17767	2974	3013	2974
q14	245	215	218	215
q15	522	455	453	453
q16	493	383	364	364
q17	963	599	587	587
q18	7508	6670	6715	6670
q19	1395	1044	1077	1044
q20	469	191	196	191
q21	3873	3256	3203	3203
q22	1092	978	1028	978
Total cold run time: 107699 ms
Total hot run time: 39867 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6674	6599	6594	6594
q2	322	237	229	229
q3	2896	2919	2953	2919
q4	2026	1847	1862	1847
q5	5696	5740	5755	5740
q6	207	132	129	129
q7	2276	1778	1828	1778
q8	3368	3478	3535	3478
q9	8800	9083	8922	8922
q10	3596	3514	3519	3514
q11	597	499	497	497
q12	843	609	616	609
q13	8117	3224	3188	3188
q14	298	266	275	266
q15	524	465	458	458
q16	516	460	452	452
q17	1854	1629	1632	1629
q18	8362	7814	7770	7770
q19	1709	1524	1597	1524
q20	2081	1814	1886	1814
q21	5214	5082	4940	4940
q22	1166	1090	1021	1021
Total cold run time: 67142 ms
Total hot run time: 59318 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196697 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b80612afeeaced57b77fbac9d63421f2f838554, data reload: false

query1	1281	949	916	916
query2	6332	1843	1856	1843
query3	10990	4492	4736	4492
query4	33505	23893	23583	23583
query5	4306	467	461	461
query6	288	178	183	178
query7	3997	311	330	311
query8	291	223	211	211
query9	9508	2567	2581	2567
query10	466	255	273	255
query11	17977	15110	15387	15110
query12	156	104	103	103
query13	1554	443	423	423
query14	8921	7729	7121	7121
query15	287	202	189	189
query16	8104	528	518	518
query17	1719	609	599	599
query18	2142	319	325	319
query19	381	168	171	168
query20	124	113	112	112
query21	208	107	116	107
query22	4780	4586	4505	4505
query23	35883	34432	33867	33867
query24	11408	2864	2913	2864
query25	559	415	411	411
query26	763	178	172	172
query27	2215	355	375	355
query28	6855	2145	2215	2145
query29	706	488	438	438
query30	251	165	162	162
query31	1071	800	817	800
query32	90	52	54	52
query33	769	296	297	296
query34	985	506	517	506
query35	878	709	717	709
query36	1105	947	954	947
query37	112	72	74	72
query38	4090	3938	4032	3938
query39	1533	1454	1509	1454
query40	207	102	99	99
query41	52	49	50	49
query42	124	106	104	104
query43	534	483	471	471
query44	1292	805	811	805
query45	194	168	169	168
query46	1151	723	721	721
query47	2029	1938	1947	1938
query48	477	384	402	384
query49	916	373	383	373
query50	825	416	431	416
query51	7431	7300	7270	7270
query52	102	90	94	90
query53	256	187	188	187
query54	1236	470	466	466
query55	85	80	76	76
query56	280	249	258	249
query57	1304	1231	1218	1218
query58	240	207	212	207
query59	3084	2965	2851	2851
query60	271	252	263	252
query61	109	110	108	108
query62	898	686	681	681
query63	225	190	191	190
query64	3921	659	639	639
query65	3335	3293	3293	3293
query66	774	301	304	301
query67	16079	15759	15574	15574
query68	4562	577	579	577
query69	439	262	258	258
query70	1159	1066	1076	1066
query71	369	258	249	249
query72	6041	3969	4034	3969
query73	758	341	360	341
query74	10407	8976	8867	8867
query75	3379	2631	2656	2631
query76	2666	1133	1085	1085
query77	394	268	277	268
query78	10641	9635	9527	9527
query79	1247	600	605	600
query80	1021	423	434	423
query81	556	222	219	219
query82	1102	90	93	90
query83	232	146	145	145
query84	243	85	83	83
query85	1280	316	294	294
query86	375	300	311	300
query87	4393	4278	4212	4212
query88	3632	2364	2340	2340
query89	418	292	294	292
query90	1883	184	185	184
query91	189	156	151	151
query92	60	48	53	48
query93	1366	550	554	550
query94	783	288	302	288
query95	370	262	257	257
query96	623	281	280	280
query97	3342	3225	3157	3157
query98	215	202	203	202
query99	1561	1285	1337	1285
Total cold run time: 299727 ms
Total hot run time: 196697 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b80612afeeaced57b77fbac9d63421f2f838554, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.51	0.50	0.52
query6	1.13	0.73	0.73
query7	0.02	0.03	0.02
query8	0.03	0.03	0.03
query9	0.57	0.51	0.50
query10	0.57	0.56	0.58
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.61
query14	0.77	0.80	0.81
query15	0.85	0.82	0.82
query16	0.38	0.39	0.38
query17	1.03	1.05	1.02
query18	0.24	0.22	0.23
query19	1.96	1.89	1.83
query20	0.01	0.02	0.01
query21	15.39	0.59	0.59
query22	2.53	2.64	1.34
query23	16.91	0.95	0.87
query24	3.13	0.85	2.29
query25	0.35	0.10	0.09
query26	0.48	0.14	0.14
query27	0.05	0.04	0.05
query28	9.47	0.54	0.48
query29	12.60	3.25	3.28
query30	0.24	0.06	0.06
query31	2.86	0.41	0.38
query32	3.23	0.46	0.45
query33	2.93	3.01	3.01
query34	17.14	4.57	4.52
query35	4.55	4.58	4.64
query36	0.68	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.02
query40	0.15	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.95 s
Total hot run time: 29.87 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.89% (11135/26582)
Line Coverage 32.41% (95354/294253)
Region Coverage 31.55% (49264/156135)
Branch Coverage 27.99% (25252/90202)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.98% (19671/26236)
Line Coverage 68.24% (200542/293888)
Region Coverage 66.36% (119996/180838)
Branch Coverage 59.73% (60920/101984)

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit d58eede into apache:branch-3.0 Aug 12, 2025
24 of 26 checks passed
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants