Skip to content

branch-3.1: [improve](load) do not block delta writer if memtable memory is low #42649#54665

Merged
morrySnow merged 1 commit intobranch-3.1from
auto-pick-42649-branch-3.1
Aug 14, 2025
Merged

branch-3.1: [improve](load) do not block delta writer if memtable memory is low #42649#54665
morrySnow merged 1 commit intobranch-3.1from
auto-pick-42649-branch-3.1

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 13, 2025

Cherry-picked from #42649

@kaijchen
[improve](load) do not block delta writer if memtable memory is low (#…
4b79f5b 
…42649)

## Proposed changes

Currently when the process memory usage is high, even if load memory
usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in `INSERT INTO SELECT`
statements.

This PR changes this behavior. If the total load memory usage is lower
than 5% (current default),
the load will no longer be blocked by hard limit.
@github-actions github-actions bot requested a review from morrySnow as a code owner August 13, 2025 06:34
@Thearas
Copy link
Contributor

Thearas commented Aug 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 13, 2025
@Thearas
Copy link
Contributor

Thearas commented Aug 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 13, 2025

TPC-H: Total hot run time: 32942 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b79f5b0583f9cc0fea4106ae03e8cc2af354800, data reload: false

------ Round 1 ----------------------------------
q1	17632	5609	5508	5508
q2	2039	393	275	275
q3	12156	1231	750	750
q4	10271	875	452	452
q5	8673	2394	2136	2136
q6	194	165	134	134
q7	919	755	612	612
q8	9328	1456	1211	1211
q9	5297	4975	4928	4928
q10	6798	2281	1815	1815
q11	468	293	273	273
q12	329	352	214	214
q13	17772	3605	3010	3010
q14	238	244	211	211
q15	527	481	473	473
q16	440	424	375	375
q17	579	871	360	360
q18	6679	6425	6429	6425
q19	1197	950	548	548
q20	322	339	214	214
q21	2764	2134	2005	2005
q22	1019	1051	1013	1013
Total cold run time: 105641 ms
Total hot run time: 32942 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5519	5519	5520	5519
q2	235	323	232	232
q3	2247	2645	2306	2306
q4	1368	1783	1429	1429
q5	4363	4962	5010	4962
q6	173	161	127	127
q7	2087	1995	1796	1796
q8	2636	2818	2766	2766
q9	7296	7231	7270	7231
q10	3022	3276	2706	2706
q11	576	516	487	487
q12	641	764	637	637
q13	3390	3809	3212	3212
q14	273	295	275	275
q15	510	463	471	463
q16	439	491	432	432
q17	1232	1749	1246	1246
q18	7556	7458	7237	7237
q19	767	1157	1081	1081
q20	2035	2082	1904	1904
q21	5311	4907	4488	4488
q22	1089	1055	1023	1023
Total cold run time: 52765 ms
Total hot run time: 51559 ms

@doris-robot
Copy link

doris-robot commented Aug 13, 2025

TPC-DS: Total hot run time: 191660 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b79f5b0583f9cc0fea4106ae03e8cc2af354800, data reload: false

query1	959	403	407	403
query2	6209	1908	1847	1847
query3	8683	196	196	196
query4	33707	24321	23468	23468
query5	3683	592	429	429
query6	276	184	170	170
query7	4200	497	330	330
query8	311	252	244	244
query9	9423	2624	2590	2590
query10	481	336	258	258
query11	18119	15414	15172	15172
query12	159	109	104	104
query13	1560	547	429	429
query14	9604	7338	6607	6607
query15	260	188	177	177
query16	8116	667	466	466
query17	1556	775	612	612
query18	2138	453	333	333
query19	224	187	166	166
query20	127	128	127	127
query21	203	125	106	106
query22	4564	4602	4340	4340
query23	35018	33900	34121	33900
query24	7317	2682	2705	2682
query25	547	488	445	445
query26	922	298	174	174
query27	2230	477	365	365
query28	5298	2236	2224	2224
query29	752	598	461	461
query30	239	191	169	169
query31	1007	959	871	871
query32	89	59	59	59
query33	497	367	291	291
query34	776	884	532	532
query35	798	806	734	734
query36	1032	1054	957	957
query37	107	91	64	64
query38	4069	3997	4013	3997
query39	1517	1464	1488	1464
query40	209	124	109	109
query41	52	52	51	51
query42	119	105	106	105
query43	501	511	477	477
query44	1364	844	835	835
query45	190	178	171	171
query46	896	1068	675	675
query47	1998	2007	1970	1970
query48	414	420	370	370
query49	723	490	413	413
query50	667	705	424	424
query51	7282	7231	7274	7231
query52	105	99	91	91
query53	231	263	193	193
query54	551	536	477	477
query55	80	79	82	79
query56	284	275	253	253
query57	1313	1250	1204	1204
query58	236	213	218	213
query59	3048	3162	3032	3032
query60	295	275	280	275
query61	112	117	111	111
query62	829	743	706	706
query63	249	201	195	195
query64	3734	1005	658	658
query65	3377	3309	3277	3277
query66	793	404	308	308
query67	16123	15864	15735	15735
query68	7736	825	548	548
query69	499	302	261	261
query70	1190	1066	1153	1066
query71	369	295	266	266
query72	5844	3647	3770	3647
query73	643	733	362	362
query74	10247	8992	9294	8992
query75	3225	3160	2678	2678
query76	3191	1168	780	780
query77	541	352	271	271
query78	10285	10335	9613	9613
query79	4077	881	591	591
query80	761	514	423	423
query81	499	256	222	222
query82	594	116	92	92
query83	176	167	143	143
query84	286	103	84	84
query85	778	350	290	290
query86	387	326	308	308
query87	4316	4329	4294	4294
query88	5069	2415	2397	2397
query89	422	331	302	302
query90	1762	189	189	189
query91	135	147	108	108
query92	68	54	53	53
query93	3009	898	547	547
query94	682	403	309	309
query95	349	279	271	271
query96	480	608	283	283
query97	3233	3258	3182	3182
query98	220	218	207	207
query99	1581	1392	1280	1280
Total cold run time: 295206 ms
Total hot run time: 191660 ms

@doris-robot
Copy link

doris-robot commented Aug 13, 2025

ClickBench: Total hot run time: 28.42 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b79f5b0583f9cc0fea4106ae03e8cc2af354800, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.11	0.11
query5	0.50	0.51	0.53
query6	1.12	0.74	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.55	0.55	0.58
query11	0.13	0.10	0.10
query12	0.13	0.11	0.11
query13	0.63	0.59	0.59
query14	0.77	0.80	0.78
query15	0.85	0.84	0.82
query16	0.39	0.39	0.39
query17	1.07	1.01	1.05
query18	0.24	0.22	0.22
query19	1.88	1.85	1.83
query20	0.01	0.01	0.01
query21	15.39	0.91	0.58
query22	0.74	0.81	0.70
query23	15.03	1.46	0.58
query24	3.12	0.48	0.77
query25	0.11	0.07	0.06
query26	0.45	0.16	0.13
query27	0.04	0.06	0.04
query28	12.59	1.07	0.43
query29	12.60	3.91	3.25
query30	0.25	0.09	0.09
query31	2.83	0.59	0.38
query32	3.23	0.54	0.46
query33	3.00	2.99	3.09
query34	16.34	5.23	4.51
query35	4.60	4.61	4.54
query36	0.65	0.49	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 102.3 s
Total hot run time: 28.42 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 45.45% (12669/27874)
Line Coverage 36.31% (113027/311294)
Region Coverage 35.35% (58438/165293)
Branch Coverage 32.58% (31823/97684)

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 13, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.24% (20997/27541)
Line Coverage 69.61% (216571/311129)
Region Coverage 67.60% (129578/191670)
Branch Coverage 61.14% (67403/110238)

@morrySnow morrySnow merged commit 1fc554c into branch-3.1 Aug 14, 2025
22 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-42649-branch-3.1 branch August 14, 2025 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @kaijchen

Comments