[fix](filecache) Fix LRU persist queue thread-safe#53046

freemandealer · 2025-07-10T07:14:33Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

1.fix LRU queue crash use after free
2.fix extra LRU queue info when 'need_to_move' flag unset
3.use concurrent queueu to record queueu change info for thread safety

ERROR: AddressSanitizer: heap-use-after-free on address 0x603005548c40 at pc 0x55f28e8c4785 bp 0x7f603582e1f0 sp 0x7f603582e1e8
READ of size 8 at 0x603005548c40 thread T201
    #0 0x55f28e8c4784 in std::_Head_base<0ul, doris::io::CacheLRULog*, false>::_Head_base<doris::io::CacheLRULog*>(doris::io::CacheLRULog*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:190:17
    #1 0x55f28e8c4784 in std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::_Tuple_impl(std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:292:2
    #2 0x55f28e8c4784 in std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::tuple(std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:1079:17
    #3 0x55f28e8c4784 in std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::uniq_ptr_impl(std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:162:9
    #4 0x55f28e8c4784 in std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>::uniq_ptr_data(std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:211:7
    #5 0x55f28e8c4784 in std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::unique_ptr(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:327:7
    #6 0x55f28e8c4784 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:40:20
    #7 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24
    #8 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18
    #9 0x7f61f1842608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #10 0x7f61f1aef132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

0x603005548c40 is located 16 bytes inside of 24-byte region [0x603005548c30,0x603005548c48)
freed by thread T201 here:
    #0 0x55f28e51680d in operator delete(void*) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x3975a80d) (BuildId: 8b6ba6101e736655)
    #1 0x55f28e8c3ce0 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::pop_front() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1198:15
    #2 0x55f28e8c3ce0 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:41:19
    #3 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24
    #4 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18

previously allocated by thread T607 (CumuCompactionT) here:
    #0 0x55f28e515fad in operator new(unsigned long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x39759fad) (BuildId: 8b6ba6101e736655)
    #1 0x55f28e8c660d in __gnu_cxx::new_allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long, void const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27
    #2 0x55f28e8c660d in std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32
    #3 0x55f28e8c660d in std::allocator_traits<std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>>::allocate(std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>&, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20
    #4 0x55f28e8c660d in std::__cxx11::_List_base<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_get_node() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:442:16
    #5 0x55f28e8c660d in std::List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>* std::_cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_create_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:634:21
    #6 0x55f28e8c660d in void std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_insert<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::_List_iterator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>, std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1911:18
    #7 0x55f28e8c3522 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::push_back(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1217:15
    #8 0x55f28e8c3522 in doris::io::LRUQueueRecorder::record_queue_event(doris::io::FileCacheType, doris::io::CacheLRULogType, doris::io::UInt128Wrapper, unsigned long, unsigned long) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:29:15
    #9 0x55f28e82f09b in doris::io::BlockFileCache::use_cell(doris::io::BlockFileCache::FileBlockCell const&, std::__cxx11::list<std::shared_ptr<doris::io::FileBlock>, std::allocator<std::shared_ptr<doris::io::FileBlock>>>*, bool, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:380:20
    #10 0x55f28e833d1b in doris::io::BlockFileCache::get_impl[abi:cxx11](doris::io::UInt128Wrapper const&, doris::io::CacheContext const&, doris::io::FileBlock::Range const&, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:572:13
    #11 0x55f28e83b4ef in doris::io::BlockFileCache::get_or_set(doris::io::UInt128Wrapper const&, unsigned long, unsigned long, doris::io::CacheContext&) /root/doris/be/src/io/cache/block_file_cache.cpp:762:27
    #12 0x55f28e7ffcee in doris::io::CachedRemoteFileReader::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/cache/cached_remote_file_reader.cpp:191:21
    #13 0x55f28e7f8017 in doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/fs/file_reader.cpp:34:17

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

1.fix LRU queue crash use after free 2.fix extra LRU queue info when 'need_to_move' flag unset 3.use concurrent queueu to record queueu change info for thread safety Signed-off-by: zhengyu <[email protected]>

Thearas · 2025-07-10T07:14:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

freemandealer · 2025-07-10T07:14:46Z

run buildall

github-actions · 2025-07-10T07:37:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-10T07:37:23Z

PR approved by anyone and no changes requested.

doris-robot · 2025-07-10T08:21:01Z

TPC-H: Total hot run time: 33250 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4452a73cb673742632aeef8f8a5baf9db245b6d, data reload: false

------ Round 1 ----------------------------------
q1	17590	5274	4975	4975
q2	1962	299	192	192
q3	10360	1317	735	735
q4	10277	1013	537	537
q5	7926	2450	2289	2289
q6	173	158	125	125
q7	916	742	610	610
q8	9315	1318	1081	1081
q9	6820	5074	5106	5074
q10	6888	2361	1985	1985
q11	496	321	280	280
q12	332	337	212	212
q13	17776	3747	3128	3128
q14	227	226	224	224
q15	557	487	473	473
q16	416	420	374	374
q17	623	866	364	364
q18	7527	7155	7157	7155
q19	2207	956	562	562
q20	351	335	221	221
q21	3756	3154	2360	2360
q22	359	314	294	294
Total cold run time: 106854 ms
Total hot run time: 33250 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5515	5130	5084	5084
q2	251	331	221	221
q3	2156	2730	2289	2289
q4	1364	1808	1375	1375
q5	4226	4453	4479	4453
q6	225	169	127	127
q7	2076	1980	1774	1774
q8	2714	2723	2649	2649
q9	7491	7348	7232	7232
q10	3157	3274	2854	2854
q11	569	503	490	490
q12	682	1007	657	657
q13	3559	3961	3533	3533
q14	355	299	289	289
q15	516	485	464	464
q16	449	499	445	445
q17	1181	1492	1399	1399
q18	7962	7736	7846	7736
q19	807	808	833	808
q20	2021	2096	1940	1940
q21	4905	4335	4288	4288
q22	627	565	555	555
Total cold run time: 52808 ms
Total hot run time: 50662 ms

doris-robot · 2025-07-10T08:32:36Z

TPC-DS: Total hot run time: 185686 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4452a73cb673742632aeef8f8a5baf9db245b6d, data reload: false

query1	1022	394	410	394
query2	6624	1753	1730	1730
query3	6735	216	224	216
query4	26329	23975	22992	22992
query5	4366	597	457	457
query6	298	222	207	207
query7	4622	492	293	293
query8	263	228	208	208
query9	8619	2636	2667	2636
query10	479	333	290	290
query11	15389	15013	15037	15013
query12	160	107	105	105
query13	1657	546	415	415
query14	9176	5871	5832	5832
query15	220	188	178	178
query16	7659	414	257	257
query17	1342	699	570	570
query18	2008	398	300	300
query19	199	186	169	169
query20	131	120	113	113
query21	213	118	103	103
query22	4029	4027	3953	3953
query23	34091	33059	32810	32810
query24	8421	2390	2397	2390
query25	525	472	391	391
query26	1221	263	151	151
query27	2758	508	349	349
query28	4290	2163	2128	2128
query29	732	558	433	433
query30	285	228	196	196
query31	919	843	762	762
query32	69	63	63	63
query33	569	334	286	286
query34	797	842	534	534
query35	592	644	565	565
query36	925	985	864	864
query37	115	103	78	78
query38	4246	4089	4064	4064
query39	1466	1442	1398	1398
query40	217	116	112	112
query41	57	55	59	55
query42	131	116	112	112
query43	522	500	471	471
query44	1371	828	831	828
query45	185	173	169	169
query46	891	1028	654	654
query47	1762	1769	1720	1720
query48	396	427	317	317
query49	753	498	403	403
query50	679	694	419	419
query51	4171	4182	4263	4182
query52	109	112	107	107
query53	233	260	193	193
query54	596	584	577	577
query55	98	83	82	82
query56	296	302	294	294
query57	1170	1199	1119	1119
query58	262	248	256	248
query59	2557	2652	2585	2585
query60	342	320	316	316
query61	128	126	128	126
query62	772	710	679	679
query63	225	194	193	193
query64	4330	1244	822	822
query65	4309	4170	4198	4170
query66	1064	410	317	317
query67	15811	15669	15685	15669
query68	8135	904	528	528
query69	506	317	271	271
query70	1191	1116	1033	1033
query71	474	338	312	312
query72	5517	4773	4729	4729
query73	703	623	351	351
query74	8908	8967	8672	8672
query75	3737	3176	2691	2691
query76	3604	1127	725	725
query77	806	398	309	309
query78	10883	10966	10329	10329
query79	2018	874	596	596
query80	589	517	455	455
query81	474	313	214	214
query82	184	122	106	106
query83	264	253	232	232
query84	254	100	82	82
query85	754	363	310	310
query86	375	297	277	277
query87	4476	4438	4246	4246
query88	3624	2286	2265	2265
query89	379	317	297	297
query90	1963	210	208	208
query91	143	144	108	108
query92	78	61	58	58
query93	1625	967	588	588
query94	638	313	193	193
query95	383	288	277	277
query96	497	560	278	278
query97	2686	2736	2629	2629
query98	243	204	206	204
query99	1315	1406	1281	1281
Total cold run time: 274424 ms
Total hot run time: 185686 ms

doris-robot · 2025-07-10T08:37:52Z

ClickBench: Total hot run time: 29.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4452a73cb673742632aeef8f8a5baf9db245b6d, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.05
query3	0.25	0.07	0.08
query4	1.62	0.11	0.11
query5	0.45	0.44	0.41
query6	1.18	0.66	0.68
query7	0.03	0.02	0.01
query8	0.05	0.04	0.04
query9	0.59	0.51	0.53
query10	0.57	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.61	0.63
query14	0.79	0.80	0.81
query15	0.91	0.88	0.87
query16	0.39	0.39	0.38
query17	1.05	1.08	1.07
query18	0.23	0.22	0.21
query19	1.98	1.83	1.91
query20	0.01	0.01	0.02
query21	15.37	0.88	0.55
query22	0.76	1.17	0.59
query23	15.05	1.36	0.60
query24	7.05	1.35	0.77
query25	0.50	0.21	0.09
query26	0.64	0.16	0.14
query27	0.07	0.06	0.05
query28	9.66	0.88	0.45
query29	12.57	4.08	3.40
query30	0.25	0.10	0.07
query31	2.83	0.59	0.38
query32	3.23	0.57	0.48
query33	3.06	3.12	3.12
query34	16.18	5.32	4.77
query35	4.80	4.79	4.85
query36	0.70	0.50	0.50
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.4 s
Total hot run time: 29.53 s

dataroaring

LGTM

Signed-off-by: zhengyu <[email protected]>

freemandealer · 2025-07-11T11:35:36Z

run buildall

github-actions · 2025-07-11T11:38:31Z

PR approved by at least one committer and no changes requested.

doris-robot · 2025-07-11T13:11:49Z

TPC-H: Total hot run time: 33410 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8d12d425a098a5b1661bbf01ba8376d3e4cf479, data reload: false

------ Round 1 ----------------------------------
q1	17556	5244	4999	4999
q2	1916	283	194	194
q3	10326	1294	747	747
q4	10226	1030	540	540
q5	7550	2435	2378	2378
q6	178	156	127	127
q7	923	759	598	598
q8	9296	1315	1086	1086
q9	6836	5060	5197	5060
q10	6892	2414	1963	1963
q11	504	291	280	280
q12	343	350	221	221
q13	17779	3679	3069	3069
q14	236	224	219	219
q15	552	487	476	476
q16	435	420	373	373
q17	612	897	369	369
q18	7659	7295	7207	7207
q19	1316	978	566	566
q20	344	371	222	222
q21	4176	3197	2411	2411
q22	354	329	305	305
Total cold run time: 106009 ms
Total hot run time: 33410 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5183	5139	5115	5115
q2	239	330	215	215
q3	2191	2731	2318	2318
q4	1441	1804	1340	1340
q5	4259	4087	4126	4087
q6	203	165	121	121
q7	1926	1812	1671	1671
q8	2484	2462	2399	2399
q9	6808	6687	6808	6687
q10	2986	3155	2711	2711
q11	574	518	490	490
q12	638	709	589	589
q13	3338	3638	3085	3085
q14	284	283	262	262
q15	507	470	462	462
q16	420	469	438	438
q17	1140	1491	1372	1372
q18	7489	7271	7149	7149
q19	825	828	900	828
q20	1920	1968	1830	1830
q21	4738	4350	4328	4328
q22	618	595	574	574
Total cold run time: 50211 ms
Total hot run time: 48071 ms

doris-robot · 2025-07-11T13:23:25Z

TPC-DS: Total hot run time: 187053 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8d12d425a098a5b1661bbf01ba8376d3e4cf479, data reload: false

query1	1010	389	392	389
query2	6501	1685	1674	1674
query3	6746	215	220	215
query4	26481	23823	22971	22971
query5	5103	605	438	438
query6	304	234	213	213
query7	4646	497	291	291
query8	271	227	234	227
query9	8642	2669	2678	2669
query10	497	331	271	271
query11	15502	15034	14806	14806
query12	169	109	103	103
query13	1635	518	396	396
query14	9527	5954	6100	5954
query15	206	189	161	161
query16	7632	427	269	269
query17	1447	692	594	594
query18	1987	382	298	298
query19	198	202	161	161
query20	116	114	115	114
query21	212	154	112	112
query22	4053	4168	4113	4113
query23	33993	33171	33068	33068
query24	8413	2398	2354	2354
query25	527	471	403	403
query26	1225	268	151	151
query27	2759	493	346	346
query28	4311	2141	2125	2125
query29	753	553	457	457
query30	287	218	192	192
query31	921	822	735	735
query32	79	68	65	65
query33	549	346	274	274
query34	785	841	539	539
query35	615	643	549	549
query36	934	979	875	875
query37	115	103	81	81
query38	4162	4102	4062	4062
query39	1514	1445	1452	1445
query40	213	132	111	111
query41	65	92	49	49
query42	121	110	110	110
query43	517	516	468	468
query44	1352	814	804	804
query45	174	163	159	159
query46	845	1041	632	632
query47	1752	1762	1722	1722
query48	380	418	304	304
query49	752	470	392	392
query50	658	738	422	422
query51	5416	5516	5490	5490
query52	113	109	98	98
query53	225	268	195	195
query54	584	613	516	516
query55	85	83	91	83
query56	313	278	280	278
query57	1178	1189	1088	1088
query58	262	250	249	249
query59	2562	2641	2500	2500
query60	327	318	299	299
query61	130	122	126	122
query62	791	697	651	651
query63	226	188	189	188
query64	4378	1206	884	884
query65	4300	4177	4165	4165
query66	1047	399	315	315
query67	16014	15582	15474	15474
query68	7428	874	523	523
query69	504	315	281	281
query70	1186	1094	1146	1094
query71	454	323	302	302
query72	5616	4790	4828	4790
query73	680	650	351	351
query74	9296	9064	8766	8766
query75	3192	3166	2662	2662
query76	3211	1145	749	749
query77	488	385	297	297
query78	10973	11135	10426	10426
query79	1108	817	561	561
query80	666	605	457	457
query81	468	263	218	218
query82	184	124	103	103
query83	257	251	232	232
query84	245	101	89	89
query85	746	363	318	318
query86	316	305	283	283
query87	4556	4392	4235	4235
query88	2844	2287	2252	2252
query89	370	311	280	280
query90	1780	219	209	209
query91	134	139	112	112
query92	60	60	56	56
query93	1008	942	579	579
query94	634	320	202	202
query95	373	294	294	294
query96	496	561	275	275
query97	2712	2740	2661	2661
query98	217	209	206	206
query99	1322	1403	1262	1262
Total cold run time: 273131 ms
Total hot run time: 187053 ms

doris-robot · 2025-07-11T13:28:42Z

ClickBench: Total hot run time: 29.82 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8d12d425a098a5b1661bbf01ba8376d3e4cf479, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.08	0.08
query4	1.62	0.10	0.10
query5	0.43	0.42	0.41
query6	1.16	0.66	0.67
query7	0.03	0.02	0.03
query8	0.05	0.04	0.03
query9	0.61	0.51	0.51
query10	0.57	0.56	0.58
query11	0.15	0.10	0.10
query12	0.15	0.12	0.11
query13	0.62	0.61	0.62
query14	0.81	0.81	0.83
query15	0.87	0.87	0.87
query16	0.38	0.38	0.38
query17	1.07	1.07	1.03
query18	0.22	0.21	0.21
query19	1.97	1.86	1.86
query20	0.01	0.01	0.01
query21	15.39	0.90	0.54
query22	0.76	1.22	0.61
query23	14.98	1.35	0.60
query24	6.80	1.34	1.09
query25	0.52	0.25	0.07
query26	0.61	0.17	0.14
query27	0.07	0.06	0.05
query28	10.04	0.91	0.43
query29	12.54	3.99	3.35
query30	0.25	0.10	0.06
query31	2.83	0.60	0.37
query32	3.24	0.59	0.48
query33	3.09	3.07	3.18
query34	16.05	5.41	4.81
query35	4.89	4.89	4.89
query36	0.70	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.35 s
Total hot run time: 29.82 s

hello-stephen · 2025-07-11T14:51:08Z

BE UT Coverage Report

Increment line coverage 100.00% (14/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.45% (15615/27178)
Line Coverage	46.19% (139388/301778)
Region Coverage	35.53% (104225/293376)
Branch Coverage	38.05% (45949/120749)

freemandealer · 2025-07-13T09:25:48Z

run buildall

doris-robot · 2025-07-13T10:09:12Z

TPC-H: Total hot run time: 33249 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fdaf5be055029915911672b31ee856a3fa0be000, data reload: false

------ Round 1 ----------------------------------
q1	17558	5253	5082	5082
q2	1921	277	189	189
q3	10464	1283	760	760
q4	10260	1046	535	535
q5	7502	2384	2340	2340
q6	175	157	129	129
q7	938	747	591	591
q8	9319	1292	1049	1049
q9	6759	5108	5089	5089
q10	6895	2347	1952	1952
q11	470	299	277	277
q12	340	346	222	222
q13	17786	3689	3084	3084
q14	215	231	210	210
q15	572	480	484	480
q16	424	412	365	365
q17	619	883	382	382
q18	7458	7298	7077	7077
q19	1210	934	556	556
q20	352	354	223	223
q21	3903	3182	2368	2368
q22	348	315	289	289
Total cold run time: 105488 ms
Total hot run time: 33249 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5139	5311	5126	5126
q2	249	319	229	229
q3	2196	2738	2288	2288
q4	1380	1797	1352	1352
q5	4260	4274	4500	4274
q6	213	171	127	127
q7	2091	1942	1829	1829
q8	2617	2584	2601	2584
q9	7234	7283	7280	7280
q10	3105	3398	2840	2840
q11	601	545	504	504
q12	718	822	612	612
q13	3610	4060	3343	3343
q14	292	302	300	300
q15	513	489	476	476
q16	438	496	441	441
q17	1214	1598	1362	1362
q18	7953	7731	7722	7722
q19	788	810	792	792
q20	1939	2012	1844	1844
q21	4870	4520	4332	4332
q22	651	600	572	572
Total cold run time: 52071 ms
Total hot run time: 50229 ms

doris-robot · 2025-07-13T10:20:48Z

TPC-DS: Total hot run time: 187466 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fdaf5be055029915911672b31ee856a3fa0be000, data reload: false

query1	980	376	378	376
query2	6515	1742	1752	1742
query3	6744	212	240	212
query4	26825	23713	23362	23362
query5	4326	574	450	450
query6	308	213	240	213
query7	4640	482	291	291
query8	273	224	223	223
query9	8615	2614	2626	2614
query10	474	330	273	273
query11	15533	15062	14782	14782
query12	150	108	105	105
query13	1657	525	404	404
query14	9061	5982	6024	5982
query15	213	190	165	165
query16	7579	428	245	245
query17	1342	691	557	557
query18	2006	396	309	309
query19	193	187	159	159
query20	127	118	112	112
query21	206	120	98	98
query22	4112	4065	3973	3973
query23	33928	33154	33076	33076
query24	8446	2362	2403	2362
query25	522	463	374	374
query26	1229	264	143	143
query27	2774	505	345	345
query28	4331	2110	2103	2103
query29	711	557	426	426
query30	286	229	198	198
query31	946	823	747	747
query32	68	61	63	61
query33	542	359	280	280
query34	793	826	513	513
query35	585	633	558	558
query36	950	984	859	859
query37	115	106	73	73
query38	4139	4181	4083	4083
query39	1475	1419	1430	1419
query40	209	116	101	101
query41	55	53	52	52
query42	130	109	109	109
query43	504	524	481	481
query44	1296	825	820	820
query45	175	171	206	171
query46	835	1012	638	638
query47	1770	1767	1738	1738
query48	364	403	301	301
query49	731	474	386	386
query50	633	696	410	410
query51	5482	5567	5417	5417
query52	117	104	95	95
query53	221	249	183	183
query54	565	564	513	513
query55	88	81	83	81
query56	296	295	286	286
query57	1167	1181	1130	1130
query58	264	256	258	256
query59	2680	2825	2669	2669
query60	319	311	296	296
query61	122	122	123	122
query62	817	723	657	657
query63	224	185	192	185
query64	4534	1183	828	828
query65	4319	4196	4197	4196
query66	1087	434	330	330
query67	15634	15533	15476	15476
query68	7827	909	538	538
query69	503	370	263	263
query70	1209	1113	1096	1096
query71	407	322	302	302
query72	5557	4747	4889	4747
query73	675	633	348	348
query74	8859	9227	8730	8730
query75	3221	3171	2722	2722
query76	3267	1164	737	737
query77	531	389	292	292
query78	10842	11109	10292	10292
query79	1714	875	592	592
query80	660	507	443	443
query81	472	259	231	231
query82	193	130	105	105
query83	254	247	301	247
query84	252	110	91	91
query85	801	364	314	314
query86	373	301	292	292
query87	4475	4422	4427	4422
query88	2866	2307	2317	2307
query89	379	303	283	283
query90	1884	205	202	202
query91	145	150	106	106
query92	61	60	52	52
query93	1119	949	606	606
query94	647	312	192	192
query95	371	289	278	278
query96	504	559	281	281
query97	2694	2785	2669	2669
query98	220	213	203	203
query99	1346	1409	1289	1289
Total cold run time: 272685 ms
Total hot run time: 187466 ms

doris-robot · 2025-07-13T10:26:04Z

ClickBench: Total hot run time: 29.58 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fdaf5be055029915911672b31ee856a3fa0be000, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.25	0.08	0.07
query4	1.60	0.11	0.11
query5	0.44	0.46	0.41
query6	1.15	0.68	0.67
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.60	0.50	0.50
query10	0.57	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.63	0.61	0.62
query14	0.80	0.80	0.84
query15	0.90	0.89	0.85
query16	0.39	0.39	0.40
query17	1.06	1.06	1.11
query18	0.23	0.22	0.21
query19	1.99	1.87	1.85
query20	0.02	0.01	0.01
query21	15.37	0.89	0.54
query22	0.75	1.27	1.00
query23	14.72	1.37	0.66
query24	6.97	1.22	0.35
query25	0.52	0.15	0.06
query26	0.59	0.17	0.13
query27	0.07	0.06	0.05
query28	8.98	0.92	0.46
query29	12.57	3.96	3.26
query30	0.25	0.10	0.06
query31	2.83	0.60	0.40
query32	3.25	0.56	0.46
query33	3.06	3.18	3.15
query34	16.16	5.48	4.84
query35	4.88	4.86	4.93
query36	0.69	0.49	0.49
query37	0.09	0.07	0.08
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.02	0.03
query42	0.03	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 103.31 s
Total hot run time: 29.58 s

doris-robot · 2025-07-13T11:24:48Z

BE UT Coverage Report

Increment line coverage 100.00% (14/14) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.45% (15617/27182)
Line Coverage	46.16% (139404/301971)
Region Coverage	35.48% (104157/293558)
Branch Coverage	38.03% (45944/120800)

freemandealer · 2025-07-14T02:12:10Z

run external

### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: 1.fix LRU queue crash use after free 2.fix extra LRU queue info when 'need_to_move' flag unset 3.use concurrent queueu to record queueu change info for thread safety ``` ERROR: AddressSanitizer: heap-use-after-free on address 0x603005548c40 at pc 0x55f28e8c4785 bp 0x7f603582e1f0 sp 0x7f603582e1e8 READ of size 8 at 0x603005548c40 thread T201 #0 0x55f28e8c4784 in std::_Head_base<0ul, doris::io::CacheLRULog*, false>::_Head_base<doris::io::CacheLRULog*>(doris::io::CacheLRULog*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:190:17 #1 0x55f28e8c4784 in std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::_Tuple_impl(std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:292:2 #2 0x55f28e8c4784 in std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::tuple(std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:1079:17 #3 0x55f28e8c4784 in std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::uniq_ptr_impl(std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:162:9 #4 0x55f28e8c4784 in std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>::uniq_ptr_data(std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:211:7 #5 0x55f28e8c4784 in std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::unique_ptr(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:327:7 #6 0x55f28e8c4784 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:40:20 #7 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24 #8 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18 #9 0x7f61f1842608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 #10 0x7f61f1aef132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x603005548c40 is located 16 bytes inside of 24-byte region [0x603005548c30,0x603005548c48) freed by thread T201 here: #0 0x55f28e51680d in operator delete(void*) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x3975a80d) (BuildId: 8b6ba6101e736655) #1 0x55f28e8c3ce0 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::pop_front() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1198:15 #2 0x55f28e8c3ce0 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:41:19 #3 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24 #4 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18 previously allocated by thread T607 (CumuCompactionT) here: #0 0x55f28e515fad in operator new(unsigned long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x39759fad) (BuildId: 8b6ba6101e736655) #1 0x55f28e8c660d in __gnu_cxx::new_allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long, void const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27 #2 0x55f28e8c660d in std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32 #3 0x55f28e8c660d in std::allocator_traits<std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>>::allocate(std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>&, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20 #4 0x55f28e8c660d in std::__cxx11::_List_base<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_get_node() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:442:16 #5 0x55f28e8c660d in std::List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>* std::_cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_create_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:634:21 #6 0x55f28e8c660d in void std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_insert<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::_List_iterator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>, std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1911:18 #7 0x55f28e8c3522 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::push_back(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1217:15 #8 0x55f28e8c3522 in doris::io::LRUQueueRecorder::record_queue_event(doris::io::FileCacheType, doris::io::CacheLRULogType, doris::io::UInt128Wrapper, unsigned long, unsigned long) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:29:15 #9 0x55f28e82f09b in doris::io::BlockFileCache::use_cell(doris::io::BlockFileCache::FileBlockCell const&, std::__cxx11::list<std::shared_ptr<doris::io::FileBlock>, std::allocator<std::shared_ptr<doris::io::FileBlock>>>*, bool, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:380:20 #10 0x55f28e833d1b in doris::io::BlockFileCache::get_impl[abi:cxx11](doris::io::UInt128Wrapper const&, doris::io::CacheContext const&, doris::io::FileBlock::Range const&, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:572:13 #11 0x55f28e83b4ef in doris::io::BlockFileCache::get_or_set(doris::io::UInt128Wrapper const&, unsigned long, unsigned long, doris::io::CacheContext&) /root/doris/be/src/io/cache/block_file_cache.cpp:762:27 #12 0x55f28e7ffcee in doris::io::CachedRemoteFileReader::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/cache/cached_remote_file_reader.cpp:191:21 #13 0x55f28e7f8017 in doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/fs/file_reader.cpp:34:17 ``` ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [x] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Signed-off-by: zhengyu <[email protected]>

…#53220) Cherry-picked from #53046 Signed-off-by: zhengyu <[email protected]> Co-authored-by: zhengyu <[email protected]>

freemandealer · 2025-07-15T15:23:59Z

this pr cannot be picked to branch-3.0 because its pre-PR #52807 is not going to merge some how

### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: 1.fix LRU queue crash use after free 2.fix extra LRU queue info when 'need_to_move' flag unset 3.use concurrent queueu to record queueu change info for thread safety ``` ERROR: AddressSanitizer: heap-use-after-free on address 0x603005548c40 at pc 0x55f28e8c4785 bp 0x7f603582e1f0 sp 0x7f603582e1e8 READ of size 8 at 0x603005548c40 thread T201 #0 0x55f28e8c4784 in std::_Head_base<0ul, doris::io::CacheLRULog*, false>::_Head_base<doris::io::CacheLRULog*>(doris::io::CacheLRULog*&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:190:17 #1 0x55f28e8c4784 in std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::_Tuple_impl(std::_Tuple_impl<0ul, doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:292:2 #2 0x55f28e8c4784 in std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>::tuple(std::tuple<doris::io::CacheLRULog*, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:1079:17 apache#3 0x55f28e8c4784 in std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::uniq_ptr_impl(std::_uniq_ptr_impl<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:162:9 apache#4 0x55f28e8c4784 in std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>::uniq_ptr_data(std::_uniq_ptr_data<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>, true, true>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:211:7 apache#5 0x55f28e8c4784 in std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>::unique_ptr(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:327:7 apache#6 0x55f28e8c4784 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:40:20 apache#7 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24 apache#8 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18 apache#9 0x7f61f1842608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8 apache#10 0x7f61f1aef132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95 0x603005548c40 is located 16 bytes inside of 24-byte region [0x603005548c30,0x603005548c48) freed by thread T201 here: #0 0x55f28e51680d in operator delete(void*) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x3975a80d) (BuildId: 8b6ba6101e736655) #1 0x55f28e8c3ce0 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::pop_front() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1198:15 #2 0x55f28e8c3ce0 in doris::io::LRUQueueRecorder::replay_queue_event(doris::io::FileCacheType) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:41:19 apache#3 0x55f28e82d620 in doris::io::BlockFileCache::run_background_lru_log_replay() /root/doris/be/src/io/cache/block_file_cache.cpp:2242:24 apache#4 0x55f2cdc2720f in execute_native_thread_routine /data/gcc-11.1.0/build/x86_64-pc-linux-gnu/libstdc+-v3/src/c11/../../../../../libstdc-v3/src/c+11/thread.cc:82:18 previously allocated by thread T607 (CumuCompactionT) here: #0 0x55f28e515fad in operator new(unsigned long) (/home/work/unlimit_teamcity/TeamCity/Agents/20250708205944agent_172.16.0.48_1/work/60183217f6ee2a9c/output/be/lib/doris_be+0x39759fad) (BuildId: 8b6ba6101e736655) #1 0x55f28e8c660d in __gnu_cxx::new_allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long, void const*) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:121:27 #2 0x55f28e8c660d in std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::allocate(unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:173:32 apache#3 0x55f28e8c660d in std::allocator_traits<std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>>::allocate(std::allocator<std::_List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>&, unsigned long) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:460:20 apache#4 0x55f28e8c660d in std::__cxx11::_List_base<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_get_node() /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:442:16 apache#5 0x55f28e8c660d in std::List_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>* std::_cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_create_node<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:634:21 apache#6 0x55f28e8c660d in void std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::_M_insert<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>(std::_List_iterator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>, std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1911:18 apache#7 0x55f28e8c3522 in std::__cxx11::list<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>, std::allocator<std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>>>::push_back(std::unique_ptr<doris::io::CacheLRULog, std::default_delete<doris::io::CacheLRULog>>&&) /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_list.h:1217:15 apache#8 0x55f28e8c3522 in doris::io::LRUQueueRecorder::record_queue_event(doris::io::FileCacheType, doris::io::CacheLRULogType, doris::io::UInt128Wrapper, unsigned long, unsigned long) /root/doris/be/src/io/cache/lru_queue_recorder.cpp:29:15 apache#9 0x55f28e82f09b in doris::io::BlockFileCache::use_cell(doris::io::BlockFileCache::FileBlockCell const&, std::__cxx11::list<std::shared_ptr<doris::io::FileBlock>, std::allocator<std::shared_ptr<doris::io::FileBlock>>>*, bool, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:380:20 apache#10 0x55f28e833d1b in doris::io::BlockFileCache::get_impl[abi:cxx11](doris::io::UInt128Wrapper const&, doris::io::CacheContext const&, doris::io::FileBlock::Range const&, std::lock_guard<std::mutex>&) /root/doris/be/src/io/cache/block_file_cache.cpp:572:13 apache#11 0x55f28e83b4ef in doris::io::BlockFileCache::get_or_set(doris::io::UInt128Wrapper const&, unsigned long, unsigned long, doris::io::CacheContext&) /root/doris/be/src/io/cache/block_file_cache.cpp:762:27 apache#12 0x55f28e7ffcee in doris::io::CachedRemoteFileReader::read_at_impl(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/cache/cached_remote_file_reader.cpp:191:21 apache#13 0x55f28e7f8017 in doris::io::FileReader::read_at(unsigned long, doris::Slice, unsigned long*, doris::io::IOContext const*) /root/doris/be/src/io/fs/file_reader.cpp:34:17 ``` ### Release note None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [x] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Signed-off-by: zhengyu <[email protected]>

[fix](filecache) fix LRU persist bug

c4452a7

1.fix LRU queue crash use after free 2.fix extra LRU queue info when 'need_to_move' flag unset 3.use concurrent queueu to record queueu change info for thread safety Signed-off-by: zhengyu <[email protected]>

freemandealer requested review from dataroaring and gavinchou as code owners July 10, 2025 07:14

morrySnow added dev/3.0.x dev/3.1.x p0_c labels Jul 10, 2025

gavinchou previously approved these changes Jul 10, 2025

View reviewed changes

gavinchou added the cloud label Jul 10, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 10, 2025

github-actions bot added the reviewed label Jul 10, 2025

gavinchou changed the title ~~[fix](filecache) fix LRU persist bugs~~ [fix](filecache) Fix LRU persist queue thread-safe Jul 10, 2025

dataroaring previously approved these changes Jul 11, 2025

View reviewed changes

fix queue.get() crash

a8d12d4

Signed-off-by: zhengyu <[email protected]>

freemandealer dismissed stale reviews from dataroaring and gavinchou via a8d12d4 July 11, 2025 11:34

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 11, 2025

gavinchou approved these changes Jul 11, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 11, 2025

Merge branch 'master' into fix-lru-persist

fdaf5be

koarz approved these changes Jul 14, 2025

View reviewed changes

dataroaring merged commit 05770cf into apache:master Jul 14, 2025
23 of 25 checks passed

github-actions bot added the dev/3.0.x-conflict label Jul 14, 2025

github-actions bot mentioned this pull request Jul 14, 2025

branch-3.1: [fix](filecache) Fix LRU persist queue thread-safe #53046 #53220

Merged

morrySnow pushed a commit that referenced this pull request Jul 15, 2025

branch-3.1: [fix](filecache) Fix LRU persist queue thread-safe #53046 (…

9e3067a

…#53220) Cherry-picked from #53046 Signed-off-by: zhengyu <[email protected]> Co-authored-by: zhengyu <[email protected]>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jul 15, 2025

dataroaring removed dev/3.0.x dev/3.0.x-conflict labels Sep 5, 2025

Conversation

freemandealer commented Jul 10, 2025 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jul 10, 2025

Uh oh!

freemandealer commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

github-actions bot commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

doris-robot commented Jul 10, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Jul 11, 2025

Uh oh!

github-actions bot commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

hello-stephen commented Jul 11, 2025

BE UT Coverage Report

Uh oh!

freemandealer commented Jul 13, 2025

Uh oh!

doris-robot commented Jul 13, 2025

Uh oh!

doris-robot commented Jul 13, 2025

Uh oh!

doris-robot commented Jul 13, 2025

Uh oh!

doris-robot commented Jul 13, 2025

BE UT Coverage Report

Uh oh!

freemandealer commented Jul 14, 2025

Uh oh!

Uh oh!

freemandealer commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Comments

freemandealer commented Jul 10, 2025 •

edited by morrySnow

Loading

freemandealer commented Jul 15, 2025 •

edited

Loading