Skip to content

[improve](load) do not block delta writer if memtable memory is low#42649

Merged
liaoxin01 merged 1 commit intoapache:masterfrom
kaijchen:memtable-no-low-limit
Nov 14, 2024
Merged

[improve](load) do not block delta writer if memtable memory is low#42649
liaoxin01 merged 1 commit intoapache:masterfrom
kaijchen:memtable-no-low-limit

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Oct 28, 2024 •
edited by liaoxin01
Loading

Proposed changes

Currently when the process memory usage is high, even if load memory usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in INSERT INTO SELECT statements.

This PR changes this behavior. If the total load memory usage is lower than 5% (current default),
the load will no longer be blocked by hard limit.

@doris-robot
Copy link

doris-robot commented Oct 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen
Copy link
Member Author

kaijchen commented Oct 28, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Oct 28, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Oct 28, 2024

TPC-H: Total hot run time: 41338 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91d6e42684840f97c41eaf77b2377acb2b191b11, data reload: false

------ Round 1 ----------------------------------
q1	17572	7382	7280	7280
q2	2042	163	191	163
q3	10598	1045	1065	1045
q4	10571	850	815	815
q5	7760	3010	3003	3003
q6	232	149	144	144
q7	1016	616	615	615
q8	9370	2002	2012	2002
q9	6467	6394	6398	6394
q10	7055	2413	2455	2413
q11	447	250	245	245
q12	406	225	215	215
q13	17772	3043	3000	3000
q14	249	220	213	213
q15	556	511	507	507
q16	636	588	599	588
q17	974	540	522	522
q18	7146	6693	6841	6693
q19	1348	1058	978	978
q20	471	185	191	185
q21	4007	3351	3316	3316
q22	1093	1021	1002	1002
Total cold run time: 107788 ms
Total hot run time: 41338 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7233	7245	7195	7195
q2	336	232	238	232
q3	2927	2931	2964	2931
q4	2102	1855	1804	1804
q5	5718	5731	5777	5731
q6	219	139	139	139
q7	2258	1842	1834	1834
q8	3371	3500	3493	3493
q9	8887	8939	8934	8934
q10	3556	3555	3537	3537
q11	588	494	481	481
q12	840	623	673	623
q13	9799	3244	3196	3196
q14	319	275	287	275
q15	594	522	507	507
q16	686	645	645	645
q17	1863	1641	1637	1637
q18	8406	7900	7635	7635
q19	1705	1518	1644	1518
q20	2138	1851	1892	1851
q21	5569	5452	5533	5452
q22	1116	1056	1043	1043
Total cold run time: 70230 ms
Total hot run time: 60693 ms

@doris-robot
Copy link

doris-robot commented Oct 28, 2024

TPC-DS: Total hot run time: 197376 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91d6e42684840f97c41eaf77b2377acb2b191b11, data reload: false

query1	1270	968	988	968
query2	6234	2061	2033	2033
query3	11427	4648	4840	4648
query4	34060	23600	23550	23550
query5	4965	450	430	430
query6	258	174	173	173
query7	3992	302	294	294
query8	283	219	219	219
query9	9604	2704	2673	2673
query10	500	254	268	254
query11	18173	15258	15347	15258
query12	152	105	101	101
query13	1570	431	436	431
query14	9108	7357	7267	7267
query15	260	181	182	181
query16	7934	467	500	467
query17	1502	587	572	572
query18	2178	306	299	299
query19	304	160	156	156
query20	124	114	113	113
query21	211	109	107	107
query22	4902	4465	4171	4171
query23	34710	34360	34164	34164
query24	7056	2785	2752	2752
query25	562	385	390	385
query26	713	158	159	158
query27	1981	280	280	280
query28	5207	2504	2486	2486
query29	741	421	411	411
query30	256	165	171	165
query31	1029	811	820	811
query32	77	55	53	53
query33	493	271	263	263
query34	910	537	550	537
query35	1036	894	884	884
query36	1067	955	960	955
query37	118	81	69	69
query38	4421	4282	4321	4282
query39	1471	1552	1423	1423
query40	191	93	94	93
query41	46	42	43	42
query42	105	97	98	97
query43	517	466	468	466
query44	1140	799	798	798
query45	183	170	167	167
query46	1132	703	699	699
query47	1912	1846	1846	1846
query48	399	316	326	316
query49	890	387	369	369
query50	809	380	409	380
query51	7123	6925	7102	6925
query52	97	86	84	84
query53	248	179	180	179
query54	509	383	383	383
query55	79	75	73	73
query56	245	232	244	232
query57	1292	1162	1159	1159
query58	213	199	205	199
query59	3171	2906	2918	2906
query60	272	246	251	246
query61	108	99	103	99
query62	829	665	682	665
query63	221	181	184	181
query64	3562	633	616	616
query65	3301	3217	3193	3193
query66	781	305	300	300
query67	16091	15737	15667	15667
query68	3495	588	562	562
query69	396	260	264	260
query70	1239	1168	1163	1163
query71	310	258	262	258
query72	6232	4129	4128	4128
query73	752	359	366	359
query74	9867	9115	9225	9115
query75	3414	2714	2711	2711
query76	1752	946	1006	946
query77	387	278	281	278
query78	10401	9591	9537	9537
query79	1711	606	605	605
query80	1227	448	444	444
query81	548	253	242	242
query82	956	120	158	120
query83	235	136	134	134
query84	239	70	72	70
query85	1150	290	303	290
query86	450	306	309	306
query87	4953	4848	4748	4748
query88	3428	2214	2167	2167
query89	411	291	287	287
query90	2039	187	184	184
query91	130	98	100	98
query92	67	49	47	47
query93	2555	550	535	535
query94	811	284	294	284
query95	350	252	233	233
query96	611	283	287	283
query97	2943	2783	2658	2658
query98	216	198	198	198
query99	1532	1320	1303	1303
Total cold run time: 291132 ms
Total hot run time: 197376 ms

@doris-robot
Copy link

doris-robot commented Oct 28, 2024

ClickBench: Total hot run time: 32.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91d6e42684840f97c41eaf77b2377acb2b191b11, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.41	0.40	0.40
query6	1.18	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.52
query10	0.55	0.54	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.10
query13	0.61	0.62	0.59
query14	2.73	2.72	2.80
query15	0.90	0.82	0.82
query16	0.39	0.38	0.38
query17	1.03	1.00	1.05
query18	0.20	0.20	0.20
query19	1.86	1.86	1.88
query20	0.01	0.01	0.01
query21	15.38	0.60	0.60
query22	2.47	1.87	1.50
query23	16.86	1.30	0.92
query24	3.15	1.09	1.26
query25	0.21	0.17	0.10
query26	0.58	0.13	0.14
query27	0.04	0.05	0.04
query28	10.49	1.08	1.07
query29	12.66	3.37	3.31
query30	0.25	0.06	0.06
query31	2.87	0.37	0.38
query32	3.28	0.45	0.45
query33	3.00	3.04	3.03
query34	17.00	4.47	4.49
query35	4.49	4.45	4.46
query36	0.67	0.47	0.48
query37	0.08	0.07	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.61 s
Total hot run time: 32.5 s

@doris-robot
Copy link

doris-robot commented Oct 28, 2024

TeamCity be ut coverage result:
Function Coverage: 37.47% (9718/25937)
Line Coverage: 28.73% (80603/280539)
Region Coverage: 28.16% (41688/148047)
Branch Coverage: 24.71% (21173/85682)
Coverage Report: http://coverage.selectdb-in.cc/coverage/91d6e42684840f97c41eaf77b2377acb2b191b11_91d6e42684840f97c41eaf77b2377acb2b191b11/report/index.html

liaoxin01
liaoxin01 approved these changes Nov 11, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 11, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 11, 2024

PR approved by anyone and no changes requested.

sollhui
sollhui approved these changes Nov 14, 2024
View reviewed changes
Copy link
Contributor

@sollhui sollhui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 72e223a into apache:master Nov 14, 2024
github-actions bot pushed a commit that referenced this pull request Nov 14, 2024
@kaijchen
[improve](load) do not block delta writer if memtable memory is low (#…
9dce3c9 
…42649)

## Proposed changes

Currently when the process memory usage is high, even if load memory
usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in `INSERT INTO SELECT`
statements.

This PR changes this behavior. If the total load memory usage is lower
than 5% (current default),
the load will no longer be blocked by hard limit.
@github-actions github-actions bot mentioned this pull request Nov 14, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 14, 2024
@github-actions @kaijchen
branch-3.0: [improve](load) do not block delta writer if memtable mem…
b85c394 
…ory is low #42649 (#43943)

Cherry-picked from #42649

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
kaijchen pushed a commit to kaijchen/doris that referenced this pull request Nov 18, 2024
@yiguolei @kaijchen
[improve](load) do not block delta writer if memtable memory is low (a…
b1045b4 
…pache#42649)

Currently when the process memory usage is high, even if load memory
usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in `INSERT INTO SELECT`
statements.

This PR changes this behavior. If the total load memory usage is lower
than 5% (current default),
the load will no longer be blocked by hard limit.
kaijchen added a commit to kaijchen/doris that referenced this pull request Nov 20, 2024
@kaijchen
[improve](load) do not block delta writer if memtable memory is low (a…
a154f50 
…pache#42649)

Currently when the process memory usage is high, even if load memory
usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in `INSERT INTO SELECT`
statements.

This PR changes this behavior. If the total load memory usage is lower
than 5% (current default),
the load will no longer be blocked by hard limit.
@kaijchen kaijchen mentioned this pull request Nov 20, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 21, 2024
@kaijchen
[improve](load) do not block delta writer if memtable memory is low (#…
9664b50 
…42649) (#44305)

backport #42649
kaijchen added a commit to kaijchen/doris that referenced this pull request Nov 21, 2024
@kaijchen
Revert "branch-3.0: [improve](load) do not block delta writer if memt…
9cbe033 
…able memory is low apache#42649 (apache#43943)"

This reverts commit b85c394.
dataroaring pushed a commit that referenced this pull request Nov 22, 2024
@kaijchen
Revert "branch-3.0: [improve](load) do not block delta writer if memt…
ce55752 
…able memory is low #42649 (#43943)" (#44428)

This reverts commit b85c394.
kaijchen added a commit to kaijchen/doris that referenced this pull request Aug 9, 2025
@kaijchen
[improve](load) do not block delta writer if memtable memory is low (a…
2b80612 
…pache#42649)

## Proposed changes

Currently when the process memory usage is high, even if load memory
usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in `INSERT INTO SELECT`
statements.

This PR changes this behavior. If the total load memory usage is lower
than 5% (current default),
the load will no longer be blocked by hard limit.
@kaijchen kaijchen mentioned this pull request Aug 9, 2025
Merged
dataroaring pushed a commit that referenced this pull request Aug 12, 2025
@kaijchen
[improve](load) do not block delta writer if memtable memory is low (#…
d58eede 
…42649) (#54517)

cherry-pick #42649
github-actions bot pushed a commit that referenced this pull request Aug 13, 2025
@kaijchen
[improve](load) do not block delta writer if memtable memory is low (#…
4b79f5b 
…42649)

## Proposed changes

Currently when the process memory usage is high, even if load memory
usage is low,
the load could be blocked by process memory reach hard limit.
This may cause query memory cannot be released in `INSERT INTO SELECT`
statements.

This PR changes this behavior. If the total load memory usage is lower
than 5% (current default),
the load will no longer be blocked by hard limit.
@github-actions github-actions bot mentioned this pull request Aug 13, 2025
Merged
morrySnow pushed a commit that referenced this pull request Aug 14, 2025
@github-actions @kaijchen
branch-3.1: [improve](load) do not block delta writer if memtable mem…
1fc554c 
…ory is low #42649 (#54665)

Cherry-picked from #42649

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

+1 more reviewer

@sollhui sollhui sollhui approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.8-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaijchen @doris-robot @liaoxin01 @sollhui @dataroaring @morrySnow