[Performance](clould) reduce tablet cache read in topn query#50803

HappenLee · 2025-05-12T04:01:36Z

What problem does this PR solve?

Topn lazy read only read the cache in cloud mode to reduce tablet cache read.

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-05-12T04:01:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

HappenLee · 2025-05-12T04:04:09Z

run buildall

doris-robot · 2025-05-12T05:54:26Z

TPC-H: Total hot run time: 33804 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1604bbc81efa2361542cca31e5a50cbe3af92a05, data reload: false

------ Round 1 ----------------------------------
q1	26333	5000	4973	4973
q2	2076	302	186	186
q3	10524	1278	699	699
q4	10232	1009	523	523
q5	7721	2384	2377	2377
q6	178	159	133	133
q7	913	741	616	616
q8	9305	1257	1140	1140
q9	6863	5054	5105	5054
q10	6833	2323	1885	1885
q11	491	283	267	267
q12	351	353	204	204
q13	17755	3739	3067	3067
q14	247	227	212	212
q15	528	476	481	476
q16	426	430	378	378
q17	586	873	374	374
q18	7823	7192	7092	7092
q19	1698	959	543	543
q20	328	336	223	223
q21	3937	3360	2411	2411
q22	1036	994	971	971
Total cold run time: 116184 ms
Total hot run time: 33804 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5126	5060	5058	5058
q2	237	331	229	229
q3	2143	2648	2277	2277
q4	1346	1762	1338	1338
q5	4494	4379	4374	4374
q6	220	178	127	127
q7	1967	1946	1772	1772
q8	2618	2661	2551	2551
q9	7237	7273	6976	6976
q10	3057	3189	2727	2727
q11	583	514	494	494
q12	687	766	597	597
q13	3455	3831	3312	3312
q14	272	311	277	277
q15	530	480	483	480
q16	439	497	426	426
q17	1162	1561	1407	1407
q18	7778	7585	7574	7574
q19	789	882	1065	882
q20	1986	1968	1854	1854
q21	5240	4855	4762	4762
q22	1096	1048	1031	1031
Total cold run time: 52462 ms
Total hot run time: 50525 ms

HappenLee · 2025-05-12T06:03:34Z

run buildall

hello-stephen · 2025-05-12T07:39:11Z

BE UT Coverage Report

Increment line coverage 18.18% (2/11) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.76% (14892/26709)
Line Coverage	44.57% (131758/295636)
Region Coverage	43.62% (66254/151886)
Branch Coverage	38.23% (33953/88818)

hello-stephen · 2025-05-12T10:16:00Z

BE Regression && UT Coverage Report

Increment line coverage 27.27% (3/11) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.42% (20867/26275)
Line Coverage	72.64% (214699/295572)
Region Coverage	70.81% (126323/178386)
Branch Coverage	64.56% (65482/101426)

doris-robot · 2025-05-12T12:11:11Z

TPC-H: Total hot run time: 34672 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1604bbc81efa2361542cca31e5a50cbe3af92a05, data reload: false

------ Round 1 ----------------------------------
q1	26175	5133	5044	5044
q2	2088	282	202	202
q3	10374	1233	731	731
q4	10231	1024	514	514
q5	7535	2229	2429	2229
q6	181	169	134	134
q7	907	756	616	616
q8	9301	1236	1099	1099
q9	6735	5102	5074	5074
q10	6854	2294	1906	1906
q11	495	286	280	280
q12	342	358	221	221
q13	17791	3686	3080	3080
q14	223	225	206	206
q15	524	490	488	488
q16	416	434	374	374
q17	585	859	361	361
q18	7627	7089	7177	7089
q19	1486	946	562	562
q20	346	325	219	219
q21	4403	3451	3267	3267
q22	1023	1001	976	976
Total cold run time: 115642 ms
Total hot run time: 34672 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5179	5112	5109	5109
q2	234	323	233	233
q3	2146	2628	2306	2306
q4	1346	1788	1430	1430
q5	4436	4368	4392	4368
q6	214	170	124	124
q7	2012	1947	1738	1738
q8	2558	2492	2465	2465
q9	7162	7216	7085	7085
q10	2990	3187	2771	2771
q11	573	524	503	503
q12	702	785	606	606
q13	3502	3842	3293	3293
q14	296	306	260	260
q15	525	477	479	477
q16	434	483	448	448
q17	1148	1564	1373	1373
q18	7868	7590	7388	7388
q19	805	787	893	787
q20	1995	2019	1836	1836
q21	5121	4832	4662	4662
q22	1092	1090	1031	1031
Total cold run time: 52338 ms
Total hot run time: 50293 ms

doris-robot · 2025-05-12T12:23:28Z

TPC-DS: Total hot run time: 193854 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1604bbc81efa2361542cca31e5a50cbe3af92a05, data reload: false

query1	1407	1111	1037	1037
query2	6249	1858	1849	1849
query3	10970	4605	4423	4423
query4	54130	25958	23095	23095
query5	5019	619	476	476
query6	341	218	219	218
query7	4880	521	293	293
query8	309	254	234	234
query9	5490	2660	2656	2656
query10	434	321	276	276
query11	15046	15006	14888	14888
query12	170	113	105	105
query13	1027	548	406	406
query14	10152	6367	6401	6367
query15	216	194	179	179
query16	7217	672	521	521
query17	1107	752	628	628
query18	1724	426	346	346
query19	207	195	178	178
query20	135	125	121	121
query21	208	131	109	109
query22	4333	4317	4303	4303
query23	34145	33572	33527	33527
query24	6620	2441	2464	2441
query25	458	513	411	411
query26	707	273	169	169
query27	2188	526	356	356
query28	2955	2210	2177	2177
query29	583	574	430	430
query30	270	218	188	188
query31	880	869	762	762
query32	75	67	74	67
query33	499	403	320	320
query34	793	896	571	571
query35	799	833	758	758
query36	959	1001	898	898
query37	111	99	80	80
query38	4286	4257	4207	4207
query39	1526	1851	1450	1450
query40	210	151	107	107
query41	57	55	57	55
query42	119	110	109	109
query43	512	531	494	494
query44	1390	871	863	863
query45	186	175	170	170
query46	867	1053	692	692
query47	1830	1837	1757	1757
query48	405	454	323	323
query49	703	527	427	427
query50	682	705	419	419
query51	4295	4269	4220	4220
query52	122	123	113	113
query53	246	265	193	193
query54	636	599	528	528
query55	95	87	84	84
query56	334	308	298	298
query57	1176	1199	1143	1143
query58	258	264	256	256
query59	2766	2893	2769	2769
query60	362	348	315	315
query61	135	130	132	130
query62	745	742	719	719
query63	235	193	191	191
query64	1821	1088	671	671
query65	4318	4246	4244	4244
query66	707	399	309	309
query67	15927	15516	15499	15499
query68	7013	892	538	538
query69	550	320	268	268
query70	1175	1144	1173	1144
query71	500	332	303	303
query72	5933	4778	4855	4778
query73	1397	639	360	360
query74	8915	9224	8820	8820
query75	4104	3266	2720	2720
query76	4193	1194	779	779
query77	725	395	298	298
query78	10131	10118	9433	9433
query79	2572	795	605	605
query80	580	503	463	463
query81	544	252	221	221
query82	445	133	99	99
query83	303	249	234	234
query84	294	111	87	87
query85	762	353	317	317
query86	361	304	296	296
query87	4392	4470	4331	4331
query88	3390	2348	2346	2346
query89	404	313	291	291
query90	1811	210	213	210
query91	220	153	113	113
query92	79	59	59	59
query93	2077	927	588	588
query94	671	422	311	311
query95	378	299	296	296
query96	495	568	287	287
query97	3160	3219	3157	3157
query98	231	227	209	209
query99	1436	1415	1275	1275
Total cold run time: 299005 ms
Total hot run time: 193854 ms

doris-robot · 2025-05-12T12:28:38Z

ClickBench: Total hot run time: 29.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1604bbc81efa2361542cca31e5a50cbe3af92a05, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.12	0.10
query3	0.27	0.19	0.19
query4	1.59	0.20	0.19
query5	0.61	0.61	0.61
query6	1.21	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.57	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.12	0.13
query13	0.61	0.60	0.60
query14	0.79	0.80	0.81
query15	0.89	0.86	0.87
query16	0.38	0.38	0.38
query17	1.01	1.04	1.08
query18	0.23	0.21	0.21
query19	1.98	1.74	1.87
query20	0.01	0.02	0.01
query21	15.39	0.90	0.57
query22	0.75	1.18	0.60
query23	15.01	1.37	0.65
query24	7.14	1.15	0.96
query25	0.50	0.19	0.08
query26	0.63	0.15	0.14
query27	0.05	0.05	0.05
query28	9.63	0.88	0.46
query29	12.52	4.01	3.29
query30	0.26	0.08	0.06
query31	2.82	0.59	0.39
query32	3.23	0.57	0.48
query33	3.02	3.15	3.04
query34	15.81	5.12	4.52
query35	4.48	4.51	4.48
query36	0.66	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.04	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 103.65 s
Total hot run time: 29.4 s

eldenmoon

LGTM

github-actions · 2025-05-13T02:34:58Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-13T02:35:00Z

PR approved by anyone and no changes requested.

be/src/cloud/cloud_tablet_mgr.cpp

eldenmoon

LGTM

Hastyshell

LGTM

github-actions · 2025-05-26T02:37:58Z

PR approved by at least one committer and no changes requested.

Topn lazy read only read the cache in cloud mode to reduce tablet cache read.

…ery #50803 (#51227) Cherry-picked from #50803 Co-authored-by: HappenLee <happenlee@selectdb.com>

…50803) Topn lazy read only read the cache in cloud mode to reduce tablet cache read.

Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix apache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix #50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix apache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>

…pache#5786) pick: apache#58044 Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix apache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

…pache#5887) Topn lazy read only read the cache in cloud mode to reduce tablet cache read. fix apache#50803 failed to pass force_use_only_cached to tablet_mgr Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>

[Performance](clould) reduce tablet cache read in topn query

1604bbc

HappenLee marked this pull request as ready for review May 12, 2025 15:50

HappenLee requested review from dataroaring and gavinchou as code owners May 12, 2025 15:50

eldenmoon approved these changes May 13, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 13, 2025

github-actions bot added the reviewed label May 13, 2025

eldenmoon reviewed May 13, 2025

View reviewed changes

be/src/cloud/cloud_tablet_mgr.cpp Show resolved Hide resolved

eldenmoon added the dev/3.0.x label May 13, 2025

eldenmoon self-requested a review May 13, 2025 02:44

eldenmoon approved these changes May 13, 2025

View reviewed changes

yiguolei added usercase Important user case type label p0_b labels May 13, 2025

gavinchou added cloud and removed approved Indicates a PR has been approved by one committer. labels May 26, 2025

Hastyshell approved these changes May 26, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 26, 2025

gavinchou approved these changes May 26, 2025

View reviewed changes

gavinchou merged commit af2cda4 into apache:master May 26, 2025
35 of 40 checks passed

github-actions bot pushed a commit that referenced this pull request May 26, 2025

[Performance](clould) reduce tablet cache read in topn query (#50803)

a11c723

Topn lazy read only read the cache in cloud mode to reduce tablet cache read.

github-actions bot mentioned this pull request May 26, 2025

branch-3.0: [Performance](clould) reduce tablet cache read in topn query #50803 #51227

Merged

dataroaring pushed a commit that referenced this pull request May 26, 2025

branch-3.0: [Performance](clould) reduce tablet cache read in topn qu…

59984e0

…ery #50803 (#51227) Cherry-picked from #50803 Co-authored-by: HappenLee <happenlee@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 26, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[Performance](clould) reduce tablet cache read in topn query (apache#…

d0baf7b

…50803) Topn lazy read only read the cache in cloud mode to reduce tablet cache read.

gavinchou mentioned this pull request Jun 11, 2025

Release Note 3.0.6 #51585

Closed

Hastyshell pushed a commit to Hastyshell/doris that referenced this pull request Jul 21, 2025

[Performance](clould) reduce tablet cache read in topn query (apache#…

40ef7b7

…50803) Topn lazy read only read the cache in cloud mode to reduce tablet cache read.

freemandealer mentioned this pull request Nov 14, 2025

[fix](cloud) avoid broadcast remote read in topn query #58044

Merged

16 tasks

freemandealer mentioned this pull request Nov 18, 2025

branch-3.1: [fix](cloud) avoid broadcast remote read in topn query #58044 #58112

Merged

freemandealer mentioned this pull request Nov 18, 2025

[branch-3.0][fix](cloud) avoid broadcast remote read in topn query [pick#58044] #58116

Closed

16 tasks

Conversation

HappenLee commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented May 12, 2025

Uh oh!

HappenLee commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

HappenLee commented May 12, 2025

Uh oh!

hello-stephen commented May 12, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 12, 2025

BE Regression && UT Coverage Report

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

doris-robot commented May 12, 2025

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 13, 2025

Uh oh!

github-actions bot commented May 13, 2025

Uh oh!

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Hastyshell left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Comments

HappenLee commented May 12, 2025 •

edited

Loading