Skip to content

branch-3.0: [fix](deadlock) avoid deadlock on tabletInvertedIndex #54197#54996

Merged
dataroaring merged 1 commit intobranch-3.0from
auto-pick-54197-branch-3.0
Aug 20, 2025
Merged

branch-3.0: [fix](deadlock) avoid deadlock on tabletInvertedIndex #54197#54996
dataroaring merged 1 commit intobranch-3.0from
auto-pick-54197-branch-3.0

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Aug 19, 2025

Cherry-picked from #54197

@dataroaring
[fix](deadlock) avoid deadlock on tabletInvertedIndex (#54197)
cb9cec5 
StampLock used by tabletInvertedIndex is a fair lock.

Comments from StampLock:
In particular, we use the phase-fair anti-barging rule: If an incoming
reader arrives while read lock is held but there is a queued writer,
this incoming reader is queued.

A dead lock happens like below:

ReportThread: hold TabletInvertedIndex.readLock and try to acquire
DatabaseTransactionMgr.readLock.

CommitThread: try to acquire TabletInvertedIndex.readLock.

TabletOperatorThread: try to acquire TabletInvertedIndex.writeLock.

CommitThread: hold DatabaseTransactionMgr.writeLock and waiting for
editlog's lock.

The DatabaseTransactionMgr.writeLock is held when writing editlog, so
sometimes it is time consuming and results in time consuming tablet
report.

Co-authored-by: Yongqiang YANG <yangyogqiang@selectdb.com>
@github-actions github-actions bot requested a review from dataroaring as a code owner August 19, 2025 07:43
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Aug 19, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-H: Total hot run time: 39825 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cb9cec5cd0f416b95d7fd70a0fe86bb0319a4821, data reload: false

------ Round 1 ----------------------------------
q1	17572	6843	6715	6715
q2	2030	214	196	196
q3	10525	1105	1157	1105
q4	10224	736	748	736
q5	7759	2810	2785	2785
q6	206	130	134	130
q7	963	610	595	595
q8	9369	1962	2003	1962
q9	6640	6402	6393	6393
q10	7052	2224	2332	2224
q11	458	260	257	257
q12	411	218	213	213
q13	17824	2952	3015	2952
q14	232	208	211	208
q15	512	454	466	454
q16	477	377	373	373
q17	968	589	539	539
q18	7237	6686	6724	6686
q19	1398	1112	1057	1057
q20	460	199	203	199
q21	3917	3047	3130	3047
q22	1100	999	1004	999
Total cold run time: 107334 ms
Total hot run time: 39825 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6647	6587	6561	6561
q2	329	234	233	233
q3	2947	2891	2925	2891
q4	2034	1860	1780	1780
q5	5713	5723	5733	5723
q6	211	128	134	128
q7	2194	1751	1804	1751
q8	3394	3525	3557	3525
q9	8728	8879	8846	8846
q10	3561	3511	3544	3511
q11	587	471	501	471
q12	782	591	579	579
q13	9041	3137	3152	3137
q14	299	274	282	274
q15	517	455	473	455
q16	478	428	435	428
q17	1845	1636	1619	1619
q18	8225	7798	7788	7788
q19	1651	1507	1583	1507
q20	2087	1852	1866	1852
q21	5224	4997	4916	4916
q22	1173	1056	1029	1029
Total cold run time: 67667 ms
Total hot run time: 59004 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

TPC-DS: Total hot run time: 193771 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cb9cec5cd0f416b95d7fd70a0fe86bb0319a4821, data reload: false

query1	920	399	402	399
query2	6201	1943	1945	1943
query3	8681	204	205	204
query4	33730	23285	23604	23285
query5	3609	453	443	443
query6	286	176	171	171
query7	4200	304	305	304
query8	287	220	231	220
query9	9442	2561	2559	2559
query10	477	267	262	262
query11	17793	15342	15252	15252
query12	160	104	103	103
query13	1542	430	442	430
query14	9310	7515	7506	7506
query15	301	172	185	172
query16	8100	537	498	498
query17	1770	608	598	598
query18	2174	327	339	327
query19	388	176	165	165
query20	120	116	122	116
query21	208	106	113	106
query22	4837	4525	4545	4525
query23	34949	34483	34299	34299
query24	11175	2942	2929	2929
query25	666	442	436	436
query26	1229	174	175	174
query27	2462	357	367	357
query28	7054	2169	2170	2169
query29	890	462	452	452
query30	279	161	165	161
query31	1041	832	843	832
query32	98	59	58	58
query33	773	321	320	320
query34	1049	542	522	522
query35	949	787	722	722
query36	1143	979	960	960
query37	125	65	70	65
query38	4059	3936	3995	3936
query39	1524	1454	1666	1454
query40	199	104	101	101
query41	54	51	48	48
query42	122	108	108	108
query43	553	498	508	498
query44	1279	801	813	801
query45	187	173	170	170
query46	1177	772	720	720
query47	2064	1939	1928	1928
query48	477	372	378	372
query49	886	396	420	396
query50	828	422	424	422
query51	7411	7366	7279	7279
query52	107	90	87	87
query53	266	188	185	185
query54	1420	465	471	465
query55	82	83	77	77
query56	273	247	265	247
query57	1355	1211	1231	1211
query58	249	208	218	208
query59	3313	3198	3117	3117
query60	295	271	271	271
query61	113	116	114	114
query62	861	718	682	682
query63	224	195	192	192
query64	4084	678	654	654
query65	3379	3295	3278	3278
query66	804	308	293	293
query67	16409	15799	15793	15793
query68	4612	568	563	563
query69	444	272	264	264
query70	1117	1085	1110	1085
query71	359	290	269	269
query72	6339	3973	4081	3973
query73	764	347	348	347
query74	10278	9044	9211	9044
query75	3343	2674	2671	2671
query76	2758	1157	1211	1157
query77	372	264	267	264
query78	10683	9571	9567	9567
query79	2212	598	596	596
query80	1118	434	429	429
query81	540	223	220	220
query82	613	86	88	86
query83	229	143	150	143
query84	234	80	76	76
query85	1373	297	293	293
query86	459	310	301	301
query87	4405	4214	4251	4214
query88	3721	2379	2363	2363
query89	409	291	299	291
query90	1934	185	196	185
query91	186	149	149	149
query92	73	49	53	49
query93	2403	546	537	537
query94	830	296	289	289
query95	353	261	258	258
query96	615	280	282	280
query97	3322	3202	3161	3161
query98	220	214	202	202
query99	1528	1294	1307	1294
Total cold run time: 300318 ms
Total hot run time: 193771 ms

@doris-robot
Copy link

doris-robot commented Aug 19, 2025

ClickBench: Total hot run time: 29.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cb9cec5cd0f416b95d7fd70a0fe86bb0319a4821, data reload: false

query1	0.03	0.02	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.11
query5	0.51	0.53	0.52
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.52
query10	0.57	0.55	0.56
query11	0.13	0.10	0.10
query12	0.13	0.10	0.11
query13	0.61	0.60	0.60
query14	0.79	0.80	0.79
query15	0.83	0.82	0.82
query16	0.38	0.39	0.38
query17	1.05	1.04	1.03
query18	0.23	0.22	0.22
query19	1.88	1.89	1.81
query20	0.02	0.02	0.02
query21	15.39	0.59	0.60
query22	2.42	1.81	1.75
query23	16.99	0.90	0.91
query24	2.75	1.01	0.67
query25	0.17	0.09	0.04
query26	0.50	0.14	0.14
query27	0.04	0.05	0.04
query28	11.02	0.53	0.46
query29	12.55	3.20	3.17
query30	0.25	0.06	0.06
query31	2.87	0.39	0.37
query32	3.22	0.46	0.46
query33	2.99	2.97	3.02
query34	16.95	4.54	4.53
query35	4.61	4.63	4.60
query36	0.65	0.47	0.47
query37	0.09	0.07	0.06
query38	0.04	0.03	0.03
query39	0.03	0.03	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.66 s
Total hot run time: 29.96 s

dataroaring
dataroaring approved these changes Aug 20, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit c8a4eb7 into branch-3.0 Aug 20, 2025
23 of 26 checks passed
@github-actions github-actions bot deleted the auto-pick-54197-branch-3.0 branch August 20, 2025 03:07
@gavinchou gavinchou mentioned this pull request Sep 1, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hello-stephen @doris-robot @dataroaring