Skip to content

[Opt](meta)persist segment rows in rowse meta pb#59476

Merged
dataroaring merged 5 commits intoapache:masterfrom
bobhan1:persist-segment-rows-in-rs-meta
Jan 15, 2026
Merged

[Opt](meta)persist segment rows in rowse meta pb#59476
dataroaring merged 5 commits intoapache:masterfrom
bobhan1:persist-segment-rows-in-rs-meta

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Dec 30, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1 bobhan1 force-pushed the persist-segment-rows-in-rs-meta branch 2 times, most recently from 10e2859 to 3e58e66 Compare December 30, 2025 06:37
@bobhan1 bobhan1 marked this pull request as ready for review December 30, 2025 06:38
@bobhan1 bobhan1 force-pushed the persist-segment-rows-in-rs-meta branch from 3e58e66 to d96f385 Compare December 30, 2025 06:39
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 30, 2025

run buildall

@bobhan1 bobhan1 mentioned this pull request Dec 30, 2025
Merged
16 tasks
@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 30, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.38% (1771/2231)
Line Coverage 64.67% (31318/48424)
Region Coverage 65.27% (15590/23887)
Branch Coverage 55.88% (8288/14832)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 30, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

TPC-H: Total hot run time: 35161 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d96f38537f3e4fbb7a16c64d6a79d345ab4e49a4, data reload: false

------ Round 1 ----------------------------------
q1	17604	4183	4030	4030
q2	2029	360	238	238
q3	10209	1336	751	751
q4	10204	787	315	315
q5	7529	2116	1962	1962
q6	182	168	136	136
q7	943	793	681	681
q8	9299	1394	1178	1178
q9	6873	5199	5174	5174
q10	6809	1820	1393	1393
q11	519	318	293	293
q12	682	767	603	603
q13	17785	3820	3081	3081
q14	289	321	274	274
q15	584	513	497	497
q16	694	669	636	636
q17	706	761	608	608
q18	7640	7411	8123	7411
q19	1100	1058	636	636
q20	434	408	257	257
q21	4524	4277	3980	3980
q22	1211	1111	1027	1027
Total cold run time: 107849 ms
Total hot run time: 35161 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4216	4247	4216	4216
q2	332	400	326	326
q3	2303	2845	2366	2366
q4	1441	1875	1398	1398
q5	4503	4317	4467	4317
q6	212	168	121	121
q7	1986	1986	1740	1740
q8	2562	2361	2319	2319
q9	7236	7035	7141	7035
q10	2485	2790	2258	2258
q11	558	474	445	445
q12	707	702	554	554
q13	3320	3753	3028	3028
q14	282	276	256	256
q15	518	483	476	476
q16	632	656	594	594
q17	1086	1200	1307	1200
q18	7498	7352	7163	7163
q19	845	837	858	837
q20	1869	1962	1798	1798
q21	4486	4283	4150	4150
q22	1127	1047	996	996
Total cold run time: 50204 ms
Total hot run time: 47593 ms

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

TPC-DS: Total hot run time: 174355 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d96f38537f3e4fbb7a16c64d6a79d345ab4e49a4, data reload: false

query5	5047	570	434	434
query6	322	225	212	212
query7	4224	473	265	265
query8	338	238	236	236
query9	8779	2625	2610	2610
query10	548	386	326	326
query11	15113	15140	14905	14905
query12	194	114	112	112
query13	1286	508	406	406
query14	6395	2962	2697	2697
query14_1	2630	2580	2629	2580
query15	204	191	173	173
query16	1016	488	420	420
query17	1047	667	560	560
query18	2562	428	325	325
query19	220	215	183	183
query20	122	113	112	112
query21	213	132	117	117
query22	3978	4078	4219	4078
query23	16016	15573	15204	15204
query23_1	15426	15453	15391	15391
query24	7402	1572	1187	1187
query24_1	1194	1207	1197	1197
query25	519	440	417	417
query26	1235	258	155	155
query27	2795	453	289	289
query28	4544	2191	2179	2179
query29	769	527	418	418
query30	308	244	208	208
query31	774	659	572	572
query32	76	67	68	67
query33	538	330	280	280
query34	901	864	538	538
query35	728	801	702	702
query36	815	855	871	855
query37	129	89	79	79
query38	2705	2728	2614	2614
query39	765	747	720	720
query39_1	689	714	711	711
query40	229	139	113	113
query41	65	62	65	62
query42	103	101	101	101
query43	442	451	424	424
query44	1328	766	756	756
query45	185	185	176	176
query46	857	968	612	612
query47	1413	1452	1371	1371
query48	303	319	247	247
query49	603	427	332	332
query50	626	276	206	206
query51	3773	3740	3697	3697
query52	102	110	98	98
query53	292	320	273	273
query54	297	252	241	241
query55	78	76	72	72
query56	287	285	283	283
query57	1042	1013	958	958
query58	265	267	253	253
query59	2094	2199	2135	2135
query60	322	313	288	288
query61	162	161	156	156
query62	392	342	334	334
query63	302	261	270	261
query64	4953	1306	971	971
query65	3745	3696	3656	3656
query66	1396	424	317	317
query67	15338	15315	15798	15315
query68	2803	1043	770	770
query69	434	349	308	308
query70	981	936	883	883
query71	306	294	263	263
query72	6338	5034	4878	4878
query73	533	614	317	317
query74	8760	8810	8578	8578
query75	2819	2877	2474	2474
query76	2918	1069	636	636
query77	334	364	267	267
query78	9725	9969	9215	9215
query79	1054	933	594	594
query80	661	557	488	488
query81	491	269	226	226
query82	1353	139	107	107
query83	373	251	234	234
query84	261	122	101	101
query85	891	509	447	447
query86	371	301	282	282
query87	2901	2861	2731	2731
query88	3312	2265	2247	2247
query89	385	335	326	326
query90	1817	157	143	143
query91	170	165	146	146
query92	69	66	59	59
query93	950	935	562	562
query94	459	324	295	295
query95	594	330	368	330
query96	589	464	204	204
query97	2357	2348	2274	2274
query98	232	191	201	191
query99	596	578	515	515
Total cold run time: 248505 ms
Total hot run time: 174355 ms

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

ClickBench: Total hot run time: 27.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d96f38537f3e4fbb7a16c64d6a79d345ab4e49a4, data reload: false

query1	0.05	0.05	0.05
query2	0.11	0.05	0.04
query3	0.26	0.08	0.09
query4	1.62	0.11	0.11
query5	0.28	0.25	0.24
query6	1.15	0.65	0.64
query7	0.03	0.03	0.03
query8	0.05	0.04	0.05
query9	0.58	0.51	0.48
query10	0.54	0.56	0.54
query11	0.15	0.10	0.11
query12	0.16	0.13	0.13
query13	0.62	0.60	0.59
query14	1.02	0.99	0.98
query15	0.82	0.79	0.81
query16	0.38	0.39	0.42
query17	1.05	0.99	1.06
query18	0.23	0.22	0.21
query19	1.91	1.87	1.81
query20	0.02	0.01	0.01
query21	15.47	0.30	0.13
query22	4.66	0.05	0.05
query23	15.94	0.30	0.10
query24	1.95	0.49	0.19
query25	0.12	0.09	0.08
query26	0.14	0.13	0.14
query27	0.05	0.05	0.06
query28	4.57	1.04	0.88
query29	12.57	3.96	3.14
query30	0.28	0.14	0.12
query31	2.83	0.61	0.38
query32	3.24	0.56	0.46
query33	3.01	3.08	3.04
query34	16.40	5.10	4.45
query35	4.53	4.72	5.10
query36	0.72	0.55	0.55
query37	0.11	0.07	0.08
query38	0.08	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.05	0.03	0.03
Total cold run time: 98.1 s
Total hot run time: 27.06 s

@bobhan1 bobhan1 force-pushed the persist-segment-rows-in-rs-meta branch from d96f385 to 91b5e99 Compare December 30, 2025 09:00
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

TPC-H: Total hot run time: 34296 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91b5e994ce7bbd7e727db0a0c63ca73d2e4c9763, data reload: false

------ Round 1 ----------------------------------
q1	16448	4223	4055	4055
q2	2058	382	252	252
q3	9659	1274	727	727
q4	9765	916	329	329
q5	7472	2139	1923	1923
q6	195	171	135	135
q7	944	796	666	666
q8	9085	1458	1226	1226
q9	6822	5151	5112	5112
q10	6769	1805	1411	1411
q11	506	305	286	286
q12	680	744	599	599
q13	17437	3839	3065	3065
q14	302	309	280	280
q15	579	508	509	508
q16	691	698	643	643
q17	719	739	626	626
q18	7574	7367	7197	7197
q19	1116	977	636	636
q20	415	378	249	249
q21	4147	4006	3401	3401
q22	1086	1042	970	970
Total cold run time: 104469 ms
Total hot run time: 34296 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4121	4032	4069	4032
q2	320	390	320	320
q3	2118	2622	2278	2278
q4	1309	1785	1287	1287
q5	4101	4001	4460	4001
q6	257	227	148	148
q7	2179	1988	1799	1799
q8	2561	2379	2324	2324
q9	7274	7244	7129	7129
q10	2558	2705	2433	2433
q11	560	500	472	472
q12	732	742	611	611
q13	3570	4119	3449	3449
q14	298	297	275	275
q15	534	509	489	489
q16	631	687	613	613
q17	1142	1340	1294	1294
q18	8095	7636	7831	7636
q19	950	908	939	908
q20	2007	2058	1898	1898
q21	4798	4555	4225	4225
q22	1097	1019	1019	1019
Total cold run time: 51212 ms
Total hot run time: 48640 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 30, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.38% (1771/2231)
Line Coverage 64.68% (31320/48424)
Region Coverage 65.22% (15579/23887)
Branch Coverage 55.85% (8284/14832)

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

TPC-DS: Total hot run time: 174886 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91b5e994ce7bbd7e727db0a0c63ca73d2e4c9763, data reload: false

query5	4391	571	446	446
query6	341	242	228	228
query7	4227	460	279	279
query8	341	256	239	239
query9	8771	2609	2642	2609
query10	518	370	318	318
query11	15128	15119	14936	14936
query12	170	114	116	114
query13	1271	514	400	400
query14	5505	2982	2730	2730
query14_1	2647	2676	2566	2566
query15	207	197	169	169
query16	967	475	371	371
query17	1021	669	581	581
query18	2426	442	335	335
query19	223	218	193	193
query20	120	115	117	115
query21	210	139	114	114
query22	3785	3986	3936	3936
query23	15998	15615	15215	15215
query23_1	15382	15504	15449	15449
query24	7065	1581	1196	1196
query24_1	1250	1196	1251	1196
query25	541	452	403	403
query26	1240	269	162	162
query27	2752	453	290	290
query28	4386	2212	2214	2212
query29	823	550	461	461
query30	317	243	213	213
query31	770	636	521	521
query32	81	73	70	70
query33	524	361	304	304
query34	888	886	547	547
query35	764	787	710	710
query36	860	870	802	802
query37	130	93	84	84
query38	2681	2675	2686	2675
query39	777	770	739	739
query39_1	722	711	724	711
query40	223	143	124	124
query41	71	69	66	66
query42	114	108	107	107
query43	477	477	420	420
query44	1345	765	763	763
query45	186	187	179	179
query46	870	965	613	613
query47	1374	1432	1367	1367
query48	341	362	279	279
query49	627	425	340	340
query50	666	287	208	208
query51	3716	3794	3753	3753
query52	107	113	100	100
query53	301	326	277	277
query54	296	275	274	274
query55	77	76	75	75
query56	298	297	312	297
query57	1037	1054	976	976
query58	266	261	256	256
query59	2011	2223	2175	2175
query60	344	339	313	313
query61	194	199	157	157
query62	394	364	315	315
query63	303	264	270	264
query64	4860	1306	1013	1013
query65	3792	3712	3736	3712
query66	1433	429	312	312
query67	14941	15252	15226	15226
query68	6886	1012	726	726
query69	573	345	311	311
query70	1052	941	882	882
query71	373	298	287	287
query72	6273	4799	4956	4799
query73	763	665	315	315
query74	8825	8792	8593	8593
query75	2930	2896	2509	2509
query76	3942	1053	653	653
query77	521	371	283	283
query78	9694	9902	9157	9157
query79	1257	936	630	630
query80	639	612	492	492
query81	510	265	230	230
query82	218	147	110	110
query83	267	256	242	242
query84	286	118	106	106
query85	935	517	467	467
query86	352	322	299	299
query87	2873	2895	2810	2810
query88	3253	2296	2280	2280
query89	400	360	339	339
query90	2040	161	153	153
query91	171	162	146	146
query92	81	69	61	61
query93	1050	909	569	569
query94	564	322	258	258
query95	573	336	359	336
query96	599	464	210	210
query97	2332	2368	2275	2275
query98	234	216	199	199
query99	557	583	490	490
Total cold run time: 250933 ms
Total hot run time: 174886 ms

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

ClickBench: Total hot run time: 27.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91b5e994ce7bbd7e727db0a0c63ca73d2e4c9763, data reload: false

query1	0.06	0.04	0.04
query2	0.11	0.05	0.04
query3	0.26	0.09	0.08
query4	1.61	0.12	0.11
query5	0.28	0.26	0.28
query6	1.14	0.68	0.66
query7	0.03	0.03	0.03
query8	0.05	0.05	0.04
query9	0.57	0.50	0.49
query10	0.54	0.55	0.54
query11	0.15	0.11	0.12
query12	0.17	0.13	0.13
query13	0.61	0.60	0.60
query14	1.00	0.99	0.99
query15	0.80	0.79	0.79
query16	0.39	0.39	0.41
query17	1.04	1.08	1.05
query18	0.23	0.22	0.21
query19	1.90	1.90	1.92
query20	0.02	0.01	0.01
query21	15.43	0.30	0.14
query22	4.87	0.06	0.05
query23	16.04	0.30	0.11
query24	0.98	0.84	0.35
query25	0.09	0.09	0.06
query26	0.14	0.15	0.14
query27	0.06	0.05	0.06
query28	3.81	1.05	0.88
query29	12.61	3.92	3.14
query30	0.28	0.14	0.12
query31	2.83	0.62	0.38
query32	3.23	0.55	0.46
query33	3.06	3.05	3.09
query34	16.61	5.12	4.46
query35	4.50	4.48	4.48
query36	0.66	0.49	0.49
query37	0.10	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.03	0.04
query40	0.18	0.15	0.13
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.74 s
Total hot run time: 27.18 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 30, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@bobhan1 bobhan1 force-pushed the persist-segment-rows-in-rs-meta branch from 91b5e99 to 9b06167 Compare December 30, 2025 10:33
@bobhan1
Copy link
Contributor Author

bobhan1 commented Dec 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.38% (1771/2231)
Line Coverage 64.67% (31317/48424)
Region Coverage 65.21% (15577/23887)
Branch Coverage 55.84% (8282/14832)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 30, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

TPC-H: Total hot run time: 35481 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b06167f482b93f1402d6aeb5b8a66757ad6d7e4, data reload: false

------ Round 1 ----------------------------------
q1	17653	4362	4097	4097
q2	2001	355	252	252
q3	10200	1297	756	756
q4	10213	876	323	323
q5	7522	2140	1925	1925
q6	202	176	144	144
q7	971	835	699	699
q8	9289	1494	1150	1150
q9	6918	5164	5217	5164
q10	6849	1785	1414	1414
q11	485	314	295	295
q12	714	760	580	580
q13	17801	3781	3084	3084
q14	286	298	281	281
q15	580	500	498	498
q16	686	695	633	633
q17	704	753	612	612
q18	7946	7628	8160	7628
q19	1150	1016	646	646
q20	449	395	257	257
q21	4566	4201	4010	4010
q22	1205	1135	1033	1033
Total cold run time: 108390 ms
Total hot run time: 35481 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4209	4300	4209
q2	336	387	306	306
q3	2365	3055	2336	2336
q4	1372	1827	1370	1370
q5	4495	4341	4323	4323
q6	216	168	135	135
q7	1940	1927	1872	1872
q8	2494	2409	2295	2295
q9	7269	7279	7211	7211
q10	2483	2517	2093	2093
q11	533	463	451	451
q12	649	722	580	580
q13	3295	3724	3013	3013
q14	262	293	258	258
q15	521	485	477	477
q16	613	638	591	591
q17	1097	1212	1307	1212
q18	7562	7389	7160	7160
q19	843	851	869	851
q20	1885	1952	1822	1822
q21	4522	4278	4126	4126
q22	1146	1046	1003	1003
Total cold run time: 50239 ms
Total hot run time: 47694 ms

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

TPC-DS: Total hot run time: 174355 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b06167f482b93f1402d6aeb5b8a66757ad6d7e4, data reload: false

query5	4806	570	431	431
query6	346	241	248	241
query7	4236	465	271	271
query8	339	265	243	243
query9	8808	2627	2624	2624
query10	520	368	318	318
query11	15378	15155	14811	14811
query12	179	111	111	111
query13	1272	493	389	389
query14	6311	3007	2721	2721
query14_1	2615	2621	2628	2621
query15	199	196	186	186
query16	999	488	468	468
query17	1069	696	587	587
query18	2617	435	349	349
query19	232	225	195	195
query20	123	120	115	115
query21	221	140	126	126
query22	4056	3890	3915	3890
query23	15986	15617	15272	15272
query23_1	15445	15405	15363	15363
query24	7504	1582	1181	1181
query24_1	1211	1192	1198	1192
query25	524	439	390	390
query26	1246	299	151	151
query27	2745	450	292	292
query28	4564	2185	2190	2185
query29	754	533	423	423
query30	307	240	212	212
query31	789	622	555	555
query32	81	70	68	68
query33	525	332	289	289
query34	913	903	525	525
query35	745	778	687	687
query36	843	892	831	831
query37	135	103	81	81
query38	2782	2688	2643	2643
query39	771	761	742	742
query39_1	714	699	708	699
query40	214	132	115	115
query41	68	64	61	61
query42	107	100	101	100
query43	439	464	404	404
query44	1328	783	745	745
query45	188	183	182	182
query46	885	966	596	596
query47	1416	1456	1308	1308
query48	317	320	250	250
query49	604	415	329	329
query50	638	270	209	209
query51	3753	3806	3744	3744
query52	109	103	96	96
query53	293	318	277	277
query54	285	250	269	250
query55	78	75	69	69
query56	276	284	274	274
query57	1008	981	968	968
query58	272	244	256	244
query59	2060	2199	2171	2171
query60	316	308	291	291
query61	160	160	153	153
query62	392	362	330	330
query63	300	265	268	265
query64	4958	1308	971	971
query65	3793	3731	3712	3712
query66	1397	428	311	311
query67	15273	15192	15150	15150
query68	3375	1049	741	741
query69	477	340	304	304
query70	1053	918	966	918
query71	352	303	272	272
query72	6273	4963	4993	4963
query73	705	611	315	315
query74	8732	8707	8573	8573
query75	2873	2887	2484	2484
query76	3815	1047	660	660
query77	512	362	276	276
query78	9896	9889	9199	9199
query79	1264	928	610	610
query80	1504	563	523	523
query81	538	261	227	227
query82	398	138	109	109
query83	362	254	243	243
query84	271	125	108	108
query85	927	521	449	449
query86	391	320	324	320
query87	2838	2834	2746	2746
query88	3264	2285	2264	2264
query89	388	350	329	329
query90	1958	153	155	153
query91	182	162	139	139
query92	70	69	60	60
query93	1067	933	564	564
query94	635	324	291	291
query95	570	329	302	302
query96	598	460	209	209
query97	2310	2380	2287	2287
query98	209	204	201	201
query99	593	590	521	521
Total cold run time: 251306 ms
Total hot run time: 174355 ms

@doris-robot
Copy link

doris-robot commented Dec 30, 2025

ClickBench: Total hot run time: 27.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9b06167f482b93f1402d6aeb5b8a66757ad6d7e4, data reload: false

query1	0.06	0.05	0.05
query2	0.15	0.07	0.06
query3	0.33	0.08	0.08
query4	1.61	0.09	0.09
query5	0.26	0.24	0.25
query6	1.14	0.64	0.64
query7	0.03	0.02	0.02
query8	0.07	0.06	0.07
query9	0.58	0.50	0.50
query10	0.54	0.55	0.56
query11	0.26	0.14	0.14
query12	0.26	0.14	0.15
query13	0.63	0.62	0.60
query14	1.01	1.00	1.00
query15	0.87	0.82	0.83
query16	0.39	0.40	0.38
query17	1.03	1.03	0.98
query18	0.23	0.21	0.21
query19	1.91	1.82	1.84
query20	0.03	0.02	0.01
query21	15.40	0.28	0.24
query22	4.96	0.10	0.09
query23	15.42	0.39	0.22
query24	2.47	0.45	0.28
query25	0.10	0.09	0.09
query26	0.18	0.18	0.17
query27	0.09	0.08	0.10
query28	3.66	1.12	0.96
query29	12.56	4.01	3.21
query30	0.32	0.12	0.10
query31	2.80	0.64	0.42
query32	3.24	0.60	0.49
query33	3.00	3.05	3.01
query34	17.07	5.08	4.45
query35	4.48	4.50	4.55
query36	0.62	0.49	0.48
query37	0.24	0.08	0.08
query38	0.21	0.06	0.06
query39	0.07	0.05	0.05
query40	0.19	0.18	0.16
query41	0.13	0.06	0.07
query42	0.08	0.05	0.05
query43	0.06	0.06	0.05
Total cold run time: 98.74 s
Total hot run time: 27.69 s

@bobhan1 bobhan1 force-pushed the persist-segment-rows-in-rs-meta branch from 9b06167 to 3796d2c Compare December 31, 2025 07:25
@doris-robot
Copy link

doris-robot commented Jan 7, 2026

BE UT Coverage Report

Increment line coverage 74.62% (97/130) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.25% (18983/35651)
Line Coverage 39.20% (175844/448542)
Region Coverage 33.76% (136218/403519)
Branch Coverage 34.73% (58780/169247)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 7, 2026

BE Regression && UT Coverage Report

Increment line coverage 85.61% (113/132) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.22% (25170/34851)
Line Coverage 58.99% (263878/447326)
Region Coverage 53.91% (219769/407641)
Branch Coverage 55.48% (94206/169796)

@bobhan1 bobhan1 force-pushed the persist-segment-rows-in-rs-meta branch from 3dd1006 to 67dbc4d Compare January 12, 2026 02:39
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jan 12, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 12, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.57% (1784/2242)
Line Coverage 64.77% (31735/48997)
Region Coverage 65.41% (15776/24120)
Branch Coverage 56.00% (8382/14968)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 12, 2026

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Jan 12, 2026

TPC-H: Total hot run time: 32132 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67dbc4dd3c05519de5f5a444de13df604362df71, data reload: false

------ Round 1 ----------------------------------
q1	17614	4297	4063	4063
q2	2050	383	257	257
q3	10088	1273	713	713
q4	10223	873	326	326
q5	7921	2114	1853	1853
q6	210	170	136	136
q7	928	802	658	658
q8	9272	1443	1147	1147
q9	5185	4583	4637	4583
q10	6839	1824	1427	1427
q11	526	292	269	269
q12	735	734	569	569
q13	17807	3864	3067	3067
q14	283	298	275	275
q15	584	501	507	501
q16	681	680	624	624
q17	669	708	622	622
q18	6583	6482	6903	6482
q19	1162	1019	635	635
q20	430	379	256	256
q21	3216	2617	2661	2617
q22	1157	1115	1052	1052
Total cold run time: 104163 ms
Total hot run time: 32132 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4418	4297	4393	4297
q2	341	408	321	321
q3	2299	2812	2419	2419
q4	1422	1872	1378	1378
q5	4535	4388	4333	4333
q6	246	196	148	148
q7	1962	1902	1747	1747
q8	2601	2501	2402	2402
q9	7210	7099	7138	7099
q10	2432	2739	2394	2394
q11	551	494	468	468
q12	684	762	622	622
q13	3317	3777	3100	3100
q14	268	295	268	268
q15	521	501	483	483
q16	628	661	625	625
q17	1153	1376	1351	1351
q18	7572	7138	7213	7138
q19	826	799	814	799
q20	1905	1970	1805	1805
q21	4650	4311	4150	4150
q22	1062	1007	985	985
Total cold run time: 50603 ms
Total hot run time: 48332 ms

@doris-robot
Copy link

doris-robot commented Jan 12, 2026

TPC-DS: Total hot run time: 172734 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67dbc4dd3c05519de5f5a444de13df604362df71, data reload: false

query5	4483	604	455	455
query6	331	223	211	211
query7	4213	446	264	264
query8	332	250	228	228
query9	8758	2680	2679	2679
query10	531	393	325	325
query11	15284	15179	14825	14825
query12	184	114	115	114
query13	1264	486	402	402
query14	6223	3060	2782	2782
query14_1	2684	2621	2677	2621
query15	199	191	175	175
query16	991	473	450	450
query17	1120	688	567	567
query18	2447	439	340	340
query19	225	225	197	197
query20	125	123	118	118
query21	217	142	119	119
query22	4263	4334	4330	4330
query23	15922	15544	15287	15287
query23_1	15334	15512	15402	15402
query24	7451	1587	1161	1161
query24_1	1181	1176	1207	1176
query25	564	469	437	437
query26	1265	272	160	160
query27	2752	450	290	290
query28	4526	2157	2139	2139
query29	803	552	469	469
query30	310	243	216	216
query31	827	637	554	554
query32	78	73	70	70
query33	541	336	289	289
query34	879	863	506	506
query35	721	824	678	678
query36	861	871	756	756
query37	123	90	80	80
query38	2730	2745	2625	2625
query39	761	738	731	731
query39_1	714	718	718	718
query40	215	131	114	114
query41	65	62	62	62
query42	104	99	106	99
query43	442	470	421	421
query44	1316	724	715	715
query45	186	184	173	173
query46	848	940	586	586
query47	1448	1452	1330	1330
query48	307	331	237	237
query49	603	407	332	332
query50	632	270	200	200
query51	3746	3830	3812	3812
query52	104	106	95	95
query53	294	321	281	281
query54	300	245	244	244
query55	78	72	69	69
query56	288	278	299	278
query57	1038	977	926	926
query58	302	251	250	250
query59	2040	2173	2022	2022
query60	314	313	294	294
query61	169	159	162	159
query62	392	347	322	322
query63	302	265	271	265
query64	4910	1298	978	978
query65	3787	3748	3777	3748
query66	1435	426	305	305
query67	15365	15158	14848	14848
query68	7153	1003	706	706
query69	500	350	312	312
query70	1050	947	951	947
query71	373	293	275	275
query72	6063	3427	3448	3427
query73	760	712	300	300
query74	8823	8819	8577	8577
query75	2820	2818	2435	2435
query76	3350	1051	619	619
query77	525	357	283	283
query78	9588	9809	9219	9219
query79	1245	906	568	568
query80	635	575	462	462
query81	526	267	227	227
query82	208	147	110	110
query83	258	252	243	243
query84	260	119	100	100
query85	934	528	466	466
query86	382	307	271	271
query87	2847	2842	2725	2725
query88	3126	2213	2218	2213
query89	387	363	339	339
query90	2123	148	139	139
query91	170	161	141	141
query92	77	66	58	58
query93	1118	907	535	535
query94	582	308	288	288
query95	573	375	306	306
query96	575	451	201	201
query97	2330	2368	2285	2285
query98	228	198	203	198
query99	592	600	524	524
Total cold run time: 252332 ms
Total hot run time: 172734 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 12, 2026

BE UT Coverage Report

Increment line coverage 74.62% (97/130) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.04% (18990/35804)
Line Coverage 39.11% (176044/450115)
Region Coverage 33.68% (136301/404673)
Branch Coverage 34.70% (58907/169758)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 12, 2026

BE Regression && UT Coverage Report

Increment line coverage 85.61% (113/132) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.89% (25925/35084)
Line Coverage 61.30% (275562/449538)
Region Coverage 56.19% (229909/409196)
Branch Coverage 58.10% (99079/170521)

liaoxin01
liaoxin01 approved these changes Jan 13, 2026
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2026

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 13, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2026

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jan 15, 2026
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 090cd96 into apache:master Jan 15, 2026
26 of 28 checks passed
bobhan1 added a commit to bobhan1/doris that referenced this pull request Jan 20, 2026
@bobhan1
[Opt](meta)persist segment rows in rowse meta pb (apache#59476)
7ad6366
@bobhan1 bobhan1 mentioned this pull request Jan 20, 2026
Merged
yiguolei pushed a commit that referenced this pull request Feb 2, 2026
@bobhan1
branch-4.0: [Opt](meta)persist segment rows in rowse meta pb (#59476) (…
2dfa55c 
…#60058)

pick #59476
dataroaring pushed a commit that referenced this pull request Feb 12, 2026
@bobhan1
[Opt](cloud) Add segment prefetcher (#59482)
71fb984 
### What problem does this PR solve?

### Cold run test result
- Environment: 1 FE + 2 BE(aliyun,
[ecs.i4g.16xlarge](https://help.aliyun.com/zh/ecs/user-guide/instance-families-with-local-ssds#i4g),
2.7 GHz Intel® Xeon® Scalable (Ice Lake) processors with
high-performance local NVMe SSDs)
    - Network baseline bandwidth **32 Gbit/s** on each machine
- Duplicate table, 16 buckets, 6178831001 rows, data size is 396.837 GB
- Cold run means **query without any doris page, OS page cache or local
disk file cache, all data will be fetched from S3**.
- Query: `select count(user_id) from test_dup3 where env_id % 10 = 0 and
user_id%10=0 and time % 10 =0 and event_id % 10 = 0 and length(domain) %
10 = 0 and length(path) % 10 = 0 and length(query)%10=0;`
- For this query, each BE will download **114GB** data to local disk
file cache from S3
- 8 instance * 48 scanner = 384 scanner on each BE, and scanner thread
pool size is set to 48


||no prefetch|prefetch_block_size=2|
|---|---|---|
| query time | 2min47sec | 39sec273ms |
| network bandwidth</br>(on one BE) | <img width="960" height="720"
alt="slow"
src="https://github.com/user-attachments/assets/73a5d621-dc9c-4885-b336-9975230a40c9"
/>| <img width="960" height="720" alt="fast"
src="https://github.com/user-attachments/assets/0e147c4a-7b13-4eaa-a6ea-a8c171ce1c52"
/>|


based on #59476
### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.4-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@bobhan1 @hello-stephen @doris-robot @liaoxin01 @dataroaring @yiguolei