Skip to content

[fix](cloud) Fix cloud balance warm up lost idx file#57114

Merged
dataroaring merged 2 commits intoapache:masterfrom
deardeng:fix-balance-warm-up-idx
Oct 22, 2025
Merged

[fix](cloud) Fix cloud balance warm up lost idx file#57114
dataroaring merged 2 commits intoapache:masterfrom
deardeng:fix-balance-warm-up-idx

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Oct 17, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Oct 17, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng
Copy link
Contributor Author

deardeng commented Oct 17, 2025

run buildall

gavinchou
gavinchou reviewed Oct 17, 2025
View reviewed changes
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 17, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@deardeng deardeng force-pushed the fix-balance-warm-up-idx branch from 9cf6c9c to 10cb1ca Compare October 17, 2025 12:24
@deardeng
Copy link
Contributor Author

deardeng commented Oct 17, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 17, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.04% (29052/43334)
Region Coverage 67.32% (14370/21346)
Branch Coverage 57.70% (7639/13240)

@doris-robot
Copy link

doris-robot commented Oct 17, 2025

TPC-H: Total hot run time: 36694 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 10cb1ca6a825eef039b29adee449757da905c4cf, data reload: false

------ Round 1 ----------------------------------
q1	17642	5354	5235	5235
q2	1960	320	187	187
q3	10311	1399	788	788
q4	10327	1096	546	546
q5	7683	1223	901	901
q6	190	175	143	143
q7	804	810	556	556
q8	9593	1140	885	885
q9	25407	4853	4914	4853
q10	9252	4962	4488	4488
q11	527	351	326	326
q12	402	463	288	288
q13	17827	3758	3126	3126
q14	423	444	412	412
q15	627	564	583	564
q16	476	467	412	412
q17	710	924	394	394
q18	7700	7237	7102	7102
q19	1092	971	615	615
q20	602	528	391	391
q21	4792	4188	4193	4188
q22	404	318	294	294
Total cold run time: 128751 ms
Total hot run time: 36694 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5245	5471	5445	5445
q2	380	503	387	387
q3	2449	3077	2614	2614
q4	1484	1958	1564	1564
q5	4730	4598	4656	4598
q6	231	173	142	142
q7	8540	8312	8262	8262
q8	3174	3040	3066	3040
q9	26075	26135	26384	26135
q10	4731	5062	4641	4641
q11	1454	1269	1281	1269
q12	699	783	703	703
q13	3519	4051	3325	3325
q14	435	427	418	418
q15	599	536	536	536
q16	655	678	642	642
q17	2528	2787	2559	2559
q18	8248	7683	7501	7501
q19	882	876	940	876
q20	2143	2117	2112	2112
q21	12409	11451	11223	11223
q22	1396	637	556	556
Total cold run time: 92006 ms
Total hot run time: 88548 ms

@doris-robot
Copy link

doris-robot commented Oct 17, 2025

ClickBench: Total hot run time: 31.07 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 10cb1ca6a825eef039b29adee449757da905c4cf, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.06
query3	0.25	0.08	0.09
query4	1.62	0.12	0.12
query5	0.29	0.26	0.26
query6	1.20	0.66	0.64
query7	0.04	0.03	0.03
query8	0.06	0.04	0.04
query9	0.67	0.53	0.52
query10	0.58	0.58	0.58
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.65	0.61	0.62
query14	1.03	1.04	1.04
query15	0.86	0.85	0.84
query16	0.41	0.39	0.41
query17	1.05	1.08	1.07
query18	0.22	0.20	0.20
query19	1.96	1.84	1.88
query20	0.01	0.01	0.02
query21	15.45	0.95	0.57
query22	0.77	1.29	0.92
query23	14.70	1.38	0.63
query24	7.20	1.62	1.15
query25	0.49	0.27	0.09
query26	0.65	0.16	0.14
query27	0.06	0.06	0.05
query28	10.35	1.38	0.95
query29	12.60	3.93	3.28
query30	0.28	0.13	0.11
query31	2.82	0.60	0.39
query32	3.23	0.56	0.47
query33	3.18	3.07	3.21
query34	16.18	5.41	4.85
query35	4.95	4.91	4.91
query36	0.69	0.53	0.51
query37	0.10	0.07	0.08
query38	0.06	0.05	0.04
query39	0.04	0.04	0.03
query40	0.18	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 105.51 s
Total hot run time: 31.07 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 17, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.54% (17833/33940)
Line Coverage 37.81% (161964/428415)
Region Coverage 32.20% (123428/383298)
Branch Coverage 33.65% (54152/160943)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 17, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/59) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.26% (23716/33280)
Line Coverage 57.68% (246972/428213)
Region Coverage 52.77% (204882/388273)
Branch Coverage 54.51% (88225/161846)

@deardeng deardeng force-pushed the fix-balance-warm-up-idx branch from 10cb1ca to 8863d29 Compare October 18, 2025 04:03
@deardeng
Copy link
Contributor Author

deardeng commented Oct 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 18, 2025

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 80.77% (1647/2039)
Line Coverage 67.05% (29062/43346)
Region Coverage 67.35% (14380/21350)
Branch Coverage 57.71% (7643/13244)

@doris-robot
Copy link

doris-robot commented Oct 18, 2025

TPC-DS: Total hot run time: 190190 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8863d29d0b5fa35ca86bf2f2e9ab27ec3ddb43e2, data reload: false

query1	1073	459	428	428
query2	6575	1748	1737	1737
query3	6757	234	229	229
query4	26930	23324	23393	23324
query5	4901	648	508	508
query6	339	272	241	241
query7	4661	513	300	300
query8	315	270	260	260
query9	8710	2587	2607	2587
query10	499	335	281	281
query11	15531	15079	15645	15079
query12	194	132	123	123
query13	1735	654	447	447
query14	11941	9639	9880	9639
query15	244	206	197	197
query16	7981	713	539	539
query17	2456	792	689	689
query18	2075	423	352	352
query19	300	232	185	185
query20	148	134	132	132
query21	211	156	129	129
query22	4717	4478	4414	4414
query23	34937	33613	34243	33613
query24	8223	2426	2457	2426
query25	577	526	480	480
query26	1217	289	183	183
query27	2676	500	361	361
query28	4294	2186	2148	2148
query29	834	634	517	517
query30	294	222	206	206
query31	927	821	735	735
query32	93	84	71	71
query33	585	435	319	319
query34	830	874	536	536
query35	789	847	763	763
query36	959	1007	892	892
query37	127	112	127	112
query38	3511	3576	3471	3471
query39	1469	1402	1424	1402
query40	221	130	117	117
query41	61	60	58	58
query42	122	115	110	110
query43	488	506	467	467
query44	1352	854	831	831
query45	184	185	175	175
query46	839	1011	636	636
query47	1732	1757	1694	1694
query48	405	428	327	327
query49	755	502	414	414
query50	661	676	415	415
query51	3889	3847	3906	3847
query52	113	120	98	98
query53	237	281	193	193
query54	585	595	529	529
query55	89	93	83	83
query56	325	355	307	307
query57	1175	1198	1113	1113
query58	280	269	281	269
query59	2535	2635	2548	2548
query60	354	349	332	332
query61	193	201	152	152
query62	793	774	664	664
query63	236	196	202	196
query64	4441	1167	822	822
query65	4065	3982	3980	3980
query66	1090	420	339	339
query67	15378	15082	14868	14868
query68	9432	945	597	597
query69	490	336	297	297
query70	1352	1272	1291	1272
query71	504	339	331	331
query72	6021	4908	4905	4905
query73	695	598	362	362
query74	8878	8824	8756	8756
query75	4470	3388	2885	2885
query76	3822	1159	746	746
query77	1005	405	325	325
query78	9589	9841	8925	8925
query79	2651	809	599	599
query80	688	604	529	529
query81	493	267	230	230
query82	427	162	137	137
query83	291	279	246	246
query84	305	109	98	98
query85	928	470	430	430
query86	367	323	309	309
query87	3733	3692	3611	3611
query88	3364	2294	2245	2245
query89	416	342	284	284
query90	2043	226	226	226
query91	170	169	136	136
query92	80	70	67	67
query93	1258	992	650	650
query94	697	459	336	336
query95	408	349	320	320
query96	503	588	287	287
query97	2933	2982	2885	2885
query98	243	217	216	216
query99	1491	1408	1309	1309
Total cold run time: 282980 ms
Total hot run time: 190190 ms

@doris-robot
Copy link

doris-robot commented Oct 18, 2025

ClickBench: Total hot run time: 31.07 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8863d29d0b5fa35ca86bf2f2e9ab27ec3ddb43e2, data reload: false

query1	0.05	0.05	0.05
query2	0.12	0.07	0.06
query3	0.31	0.07	0.07
query4	1.61	0.09	0.09
query5	0.27	0.26	0.25
query6	1.17	0.65	0.65
query7	0.04	0.02	0.03
query8	0.06	0.05	0.06
query9	0.65	0.53	0.54
query10	0.59	0.60	0.59
query11	0.25	0.14	0.13
query12	0.26	0.14	0.13
query13	0.65	0.63	0.62
query14	1.03	1.06	1.05
query15	0.97	0.88	0.87
query16	0.39	0.39	0.41
query17	1.05	1.07	1.05
query18	0.24	0.21	0.22
query19	1.97	1.87	1.84
query20	0.02	0.01	0.01
query21	15.42	1.00	0.70
query22	0.92	1.08	0.81
query23	14.71	1.57	0.77
query24	5.36	0.59	0.31
query25	0.18	0.09	0.08
query26	0.55	0.21	0.17
query27	0.09	0.10	0.09
query28	11.09	1.64	1.04
query29	12.55	4.05	3.39
query30	0.33	0.12	0.10
query31	2.83	0.62	0.44
query32	3.26	0.60	0.51
query33	3.07	3.07	3.15
query34	16.89	5.48	4.80
query35	4.86	4.87	4.86
query36	0.66	0.53	0.52
query37	0.23	0.19	0.19
query38	0.20	0.17	0.18
query39	0.05	0.05	0.05
query40	0.22	0.17	0.19
query41	0.10	0.06	0.06
query42	0.07	0.06	0.06
query43	0.06	0.05	0.05
Total cold run time: 105.4 s
Total hot run time: 31.07 s

@doris-robot
Copy link

doris-robot commented Oct 18, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.54% (17832/33940)
Line Coverage 37.80% (161955/428420)
Region Coverage 32.22% (123491/383304)
Branch Coverage 33.65% (54151/160943)

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 18, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/58) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.26% (23716/33280)
Line Coverage 57.67% (246939/428216)
Region Coverage 52.74% (204762/388282)
Branch Coverage 54.53% (88250/161848)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 20, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 20, 2025

PR approved by anyone and no changes requested.

csun5285
csun5285 approved these changes Oct 20, 2025
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@deardeng
Copy link
Contributor Author

deardeng commented Oct 20, 2025

run cloud_p0

@deardeng
Copy link
Contributor Author

deardeng commented Oct 20, 2025

run feut

zzzxl1993
zzzxl1993 approved these changes Oct 20, 2025
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 20, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/58) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.26% (23716/33280)
Line Coverage 57.67% (246939/428216)
Region Coverage 52.74% (204762/388282)
Branch Coverage 54.53% (88250/161848)

@dataroaring dataroaring merged commit 6ee5dae into apache:master Oct 22, 2025
30 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 22, 2025
@deardeng
[fix](cloud) Fix cloud balance warm up lost idx file (#57114)
75ccc6f
@github-actions github-actions bot mentioned this pull request Oct 22, 2025
Merged
yiguolei pushed a commit that referenced this pull request Oct 23, 2025
@github-actions @deardeng
branch-4.0: [fix](cloud) Fix cloud balance warm up lost idx file #57114…
c751557 
… (#57248)

Cherry-picked from #57114

Co-authored-by: deardeng <dengxin@selectdb.com>
dwdwqfwe pushed a commit to dwdwqfwe/doris that referenced this pull request Oct 24, 2025
@deardeng @dwdwqfwe
[fix](cloud) Fix cloud balance warm up lost idx file (apache#57114)
1216a70
@deardeng deardeng mentioned this pull request Oct 31, 2025
Merged
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Oct 31, 2025
@deardeng
[fix](cloud) Fix cloud balance warm up lost idx file (apache#57114)
af653ec
morrySnow pushed a commit that referenced this pull request Nov 3, 2025
@deardeng
branch-3.1: [fix](cloud) Fix cloud balance warm up lost idx file #57114…
7fb9e92 
… (#57554)

cherry pick from #57114
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

+1 more reviewer

@csun5285 csun5285 csun5285 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.3-merged dev/4.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@deardeng @Thearas @hello-stephen @doris-robot @gavinchou @zzzxl1993 @csun5285 @yiguolei @dataroaring @morrySnow

Comments