[opt](nereids) use binary search to prune partitions#44586

924060929 · 2024-11-26T02:27:01Z

What problem does this PR solve?

use binary search to speed up prune partitions.

test: datetime partition column which range from 2020-07-02 to 2024-11-13, one date for one partition, query one partition for example dt = '2024-01-02', this pr can speed up from QPS 160 to QPS 610

TODO:

support binary search for functions, like date_add(dt, 1) = '2024-01-02;
support external table

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-26T02:27:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

924060929 · 2024-11-26T02:55:36Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/MultiColumnBound.java

...core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionItemToRange.java

...ore/src/main/java/org/apache/doris/nereids/rules/expression/rules/SortedPartitionRanges.java

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java

...src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPredicateToRange.java

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java

924060929 · 2024-12-17T12:59:07Z

run buildall

924060929 · 2024-12-18T05:13:30Z

run buildall

github-actions · 2024-12-18T05:14:09Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

924060929 · 2024-12-18T13:04:05Z

run buildall

github-actions · 2024-12-18T13:04:46Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

924060929 · 2024-12-18T13:14:42Z

run buildall

github-actions · 2024-12-18T13:15:18Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

924060929 · 2024-12-18T16:35:25Z

run buildall

github-actions · 2024-12-18T16:35:42Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

924060929 · 2024-12-20T07:02:17Z

run buildall

924060929 · 2024-12-20T07:05:18Z

run buildall

doris-robot · 2024-12-20T08:21:52Z

TPC-H: Total hot run time: 39878 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 264f4363ae65f0256a42b0160f7e793b7af599d5, data reload: false

------ Round 1 ----------------------------------
q1	17714	7520	7331	7331
q2	2061	181	171	171
q3	10631	1091	1192	1091
q4	10561	754	702	702
q5	7623	2763	2680	2680
q6	246	147	144	144
q7	981	617	593	593
q8	9247	1865	1956	1865
q9	6648	6436	6427	6427
q10	6980	2310	2345	2310
q11	477	252	261	252
q12	424	227	223	223
q13	17758	3005	2928	2928
q14	251	223	219	219
q15	563	502	511	502
q16	645	608	595	595
q17	986	523	519	519
q18	7295	6684	6751	6684
q19	1335	1013	989	989
q20	459	180	184	180
q21	4147	3197	3158	3158
q22	375	329	315	315
Total cold run time: 107407 ms
Total hot run time: 39878 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7264	7476	7273	7273
q2	332	228	232	228
q3	2928	2807	2899	2807
q4	2086	1841	1846	1841
q5	5654	5640	5654	5640
q6	228	147	152	147
q7	2244	1811	1799	1799
q8	3391	3583	3498	3498
q9	8932	8959	9009	8959
q10	3618	3575	3527	3527
q11	611	539	501	501
q12	845	621	597	597
q13	11924	3167	3115	3115
q14	317	279	269	269
q15	573	508	516	508
q16	678	633	666	633
q17	1869	1629	1619	1619
q18	8395	7855	7783	7783
q19	1730	1647	1467	1467
q20	2032	1868	1875	1868
q21	5637	5610	5426	5426
q22	632	613	602	602
Total cold run time: 71920 ms
Total hot run time: 60107 ms

doris-robot · 2024-12-20T08:33:59Z

TPC-DS: Total hot run time: 196789 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 264f4363ae65f0256a42b0160f7e793b7af599d5, data reload: false

query1	1313	1000	906	906
query2	6245	2400	2318	2318
query3	11112	4858	4724	4724
query4	32801	23423	23287	23287
query5	3652	457	460	457
query6	285	191	190	190
query7	3996	300	306	300
query8	311	237	239	237
query9	9512	2721	2722	2721
query10	462	250	255	250
query11	17731	15224	15127	15127
query12	178	107	101	101
query13	1601	424	412	412
query14	9081	7544	7367	7367
query15	265	204	192	192
query16	8204	476	453	453
query17	1696	567	573	567
query18	2035	298	319	298
query19	407	156	159	156
query20	122	116	115	115
query21	205	112	104	104
query22	4759	4732	4435	4435
query23	34423	33718	33682	33682
query24	9991	2570	2520	2520
query25	593	386	408	386
query26	706	159	157	157
query27	2139	331	336	331
query28	7220	2491	2483	2483
query29	739	403	414	403
query30	245	147	152	147
query31	1066	859	840	840
query32	94	55	58	55
query33	670	360	294	294
query34	974	515	524	515
query35	884	755	761	755
query36	1130	966	950	950
query37	138	76	71	71
query38	4198	4293	4157	4157
query39	1511	1480	1452	1452
query40	206	108	101	101
query41	44	42	43	42
query42	114	105	130	105
query43	535	492	492	492
query44	1203	815	832	815
query45	190	173	166	166
query46	1181	735	731	731
query47	2077	1913	1948	1913
query48	412	333	324	324
query49	973	382	388	382
query50	841	419	408	408
query51	7314	7283	7267	7267
query52	104	95	98	95
query53	255	187	184	184
query54	1291	420	420	420
query55	83	75	88	75
query56	269	238	254	238
query57	1317	1185	1166	1166
query58	254	230	234	230
query59	3394	3059	3128	3059
query60	280	246	260	246
query61	106	105	128	105
query62	880	686	721	686
query63	223	190	194	190
query64	3687	695	654	654
query65	3294	3211	3278	3211
query66	771	316	319	316
query67	16585	15599	15563	15563
query68	4318	551	557	551
query69	428	261	248	248
query70	1215	1116	1153	1116
query71	395	252	245	245
query72	6559	4157	4081	4081
query73	777	360	362	360
query74	9690	8813	8906	8813
query75	3404	2656	2621	2621
query76	2798	1081	1144	1081
query77	436	277	282	277
query78	10198	9394	9322	9322
query79	1098	599	590	590
query80	700	429	436	429
query81	508	238	235	235
query82	245	120	120	120
query83	163	143	142	142
query84	246	83	76	76
query85	923	377	374	374
query86	361	283	287	283
query87	4468	4531	4397	4397
query88	3576	2254	2229	2229
query89	424	294	290	290
query90	2117	193	183	183
query91	138	101	104	101
query92	65	50	54	50
query93	1373	547	537	537
query94	812	280	286	280
query95	357	256	252	252
query96	621	279	280	279
query97	2862	2613	2661	2613
query98	220	205	193	193
query99	1571	1358	1322	1322
Total cold run time: 293643 ms
Total hot run time: 196789 ms

doris-robot · 2024-12-20T08:39:30Z

ClickBench: Total hot run time: 33.33 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 264f4363ae65f0256a42b0160f7e793b7af599d5, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.23	0.08	0.07
query4	1.60	0.11	0.11
query5	0.44	0.41	0.40
query6	1.17	0.66	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.61	0.52	0.51
query10	0.55	0.59	0.56
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.60	0.61	0.59
query14	2.87	2.87	2.80
query15	0.89	0.84	0.83
query16	0.39	0.37	0.38
query17	1.04	1.03	1.03
query18	0.21	0.20	0.21
query19	1.92	1.79	1.93
query20	0.01	0.02	0.01
query21	15.38	0.57	0.58
query22	2.64	2.56	2.42
query23	17.05	0.88	0.79
query24	2.91	0.97	1.23
query25	0.26	0.23	0.07
query26	0.42	0.16	0.13
query27	0.05	0.05	0.04
query28	10.65	1.10	1.07
query29	12.56	3.29	3.28
query30	0.25	0.06	0.06
query31	2.87	0.38	0.39
query32	3.23	0.48	0.46
query33	3.07	3.12	3.12
query34	16.97	4.47	4.52
query35	4.50	4.47	4.54
query36	0.66	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.93 s
Total hot run time: 33.33 s

924060929 · 2024-12-20T14:31:21Z

run buildall

doris-robot · 2024-12-20T15:11:54Z

TPC-H: Total hot run time: 40019 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 197aa64cd0977e9b1a84031c86a154153f893ac2, data reload: false

------ Round 1 ----------------------------------
q1	17587	7435	7269	7269
q2	2063	186	191	186
q3	10521	1171	1197	1171
q4	10222	783	770	770
q5	7597	2732	2660	2660
q6	242	147	148	147
q7	1006	624	630	624
q8	9254	1866	1917	1866
q9	6724	6499	6461	6461
q10	7042	2303	2319	2303
q11	471	260	258	258
q12	435	232	225	225
q13	17798	2977	2972	2972
q14	245	212	215	212
q15	548	516	494	494
q16	665	588	582	582
q17	1012	490	605	490
q18	7457	6787	6607	6607
q19	1346	964	934	934
q20	477	190	190	190
q21	4027	3335	3278	3278
q22	377	320	327	320
Total cold run time: 107116 ms
Total hot run time: 40019 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7222	7199	7216	7199
q2	328	238	230	230
q3	2959	2741	3059	2741
q4	2146	1927	1822	1822
q5	5655	5697	5707	5697
q6	227	139	138	138
q7	2235	1805	1796	1796
q8	3402	3616	3516	3516
q9	8961	8966	9056	8966
q10	3610	3589	3546	3546
q11	594	510	498	498
q12	828	632	590	590
q13	13912	3093	3067	3067
q14	299	290	289	289
q15	554	493	504	493
q16	692	624	634	624
q17	1800	1583	1604	1583
q18	7961	7451	7593	7451
q19	1716	1570	1538	1538
q20	2063	1804	1823	1804
q21	5432	5237	5297	5237
q22	658	576	563	563
Total cold run time: 73254 ms
Total hot run time: 59388 ms

924060929 · 2024-12-23T08:29:37Z

run buildall

doris-robot · 2024-12-23T08:48:59Z

TPC-H: Total hot run time: 40120 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false

------ Round 1 ----------------------------------
q1	17606	7426	7386	7386
q2	2047	178	173	173
q3	10555	1154	1154	1154
q4	10238	674	700	674
q5	7575	2714	2660	2660
q6	238	153	152	152
q7	982	639	613	613
q8	9267	1820	1919	1820
q9	6524	6435	6486	6435
q10	6972	2324	2361	2324
q11	459	255	265	255
q12	415	228	226	226
q13	17754	2904	2958	2904
q14	269	215	218	215
q15	544	512	498	498
q16	670	618	605	605
q17	987	656	542	542
q18	7285	6672	6665	6665
q19	1350	998	1047	998
q20	497	180	179	179
q21	4481	3395	3332	3332
q22	373	310	316	310
Total cold run time: 107088 ms
Total hot run time: 40120 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7285	7248	7236	7236
q2	332	230	235	230
q3	2887	2771	3065	2771
q4	2122	1942	1910	1910
q5	5588	5608	5651	5608
q6	225	139	143	139
q7	2233	1794	1810	1794
q8	3377	3534	3491	3491
q9	8977	8857	8919	8857
q10	3603	3553	3536	3536
q11	611	505	508	505
q12	814	653	614	614
q13	13181	3057	3124	3057
q14	314	266	269	266
q15	557	501	504	501
q16	676	624	642	624
q17	1818	1611	1586	1586
q18	8050	7254	7379	7254
q19	1688	1499	1587	1499
q20	2071	1810	1849	1810
q21	5476	5271	5301	5271
q22	616	578	565	565
Total cold run time: 72501 ms
Total hot run time: 59124 ms

doris-robot · 2024-12-23T09:01:03Z

TPC-DS: Total hot run time: 190521 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false

query1	957	376	370	370
query2	6526	2414	2365	2365
query3	6709	209	212	209
query4	33941	23348	23348	23348
query5	4419	462	460	460
query6	277	198	184	184
query7	4621	307	305	305
query8	297	231	236	231
query9	9486	2746	2747	2746
query10	469	284	249	249
query11	18045	15099	15107	15099
query12	149	111	106	106
query13	1671	457	435	435
query14	9923	7855	6856	6856
query15	271	179	190	179
query16	8117	448	452	448
query17	1627	602	575	575
query18	1735	309	315	309
query19	365	164	162	162
query20	126	110	113	110
query21	215	107	104	104
query22	4507	4322	4169	4169
query23	34588	34097	33877	33877
query24	11432	2463	2500	2463
query25	668	381	386	381
query26	1776	150	151	150
query27	2642	324	326	324
query28	7908	2440	2459	2440
query29	978	403	410	403
query30	302	153	151	151
query31	1013	782	833	782
query32	101	59	57	57
query33	774	295	286	286
query34	1024	520	516	516
query35	902	740	757	740
query36	1082	985	962	962
query37	267	83	78	78
query38	4085	4236	4050	4050
query39	1464	1482	1420	1420
query40	283	109	105	105
query41	51	46	50	46
query42	113	103	107	103
query43	554	511	500	500
query44	1195	799	804	799
query45	183	172	168	168
query46	1184	706	726	706
query47	1985	1836	1880	1836
query48	409	319	315	315
query49	1275	391	387	387
query50	831	381	401	381
query51	7224	6988	6926	6926
query52	105	96	88	88
query53	260	184	189	184
query54	1313	413	415	413
query55	85	75	80	75
query56	260	241	236	236
query57	1245	1127	1101	1101
query58	237	238	220	220
query59	3266	3041	2933	2933
query60	275	250	243	243
query61	113	106	105	105
query62	904	672	683	672
query63	218	189	191	189
query64	5265	670	668	668
query65	3246	3202	3237	3202
query66	1384	307	320	307
query67	15880	15574	15478	15478
query68	5722	569	606	569
query69	435	250	249	249
query70	1213	1129	1131	1129
query71	392	262	258	258
query72	6325	4131	4235	4131
query73	781	372	368	368
query74	10238	8804	8882	8804
query75	3407	2673	2677	2673
query76	3553	1130	1115	1115
query77	548	273	277	273
query78	10132	9407	9669	9407
query79	1472	624	605	605
query80	1011	435	454	435
query81	522	229	230	229
query82	804	126	119	119
query83	240	155	144	144
query84	243	75	76	75
query85	1294	314	301	301
query86	356	297	306	297
query87	4730	4468	4428	4428
query88	3472	2269	2236	2236
query89	417	301	305	301
query90	2147	193	187	187
query91	144	111	107	107
query92	67	51	54	51
query93	1132	559	548	548
query94	1091	295	288	288
query95	349	267	258	258
query96	609	287	281	281
query97	2878	2698	2667	2667
query98	214	197	195	195
query99	1584	1317	1308	1308
Total cold run time: 302324 ms
Total hot run time: 190521 ms

doris-robot · 2024-12-23T09:06:33Z

ClickBench: Total hot run time: 32.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.40	0.41
query6	1.16	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.57	0.56	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.62	0.61
query14	2.73	2.72	2.75
query15	0.90	0.82	0.83
query16	0.37	0.39	0.39
query17	1.06	1.07	1.04
query18	0.23	0.21	0.22
query19	1.95	1.82	2.00
query20	0.02	0.02	0.01
query21	15.37	0.60	0.60
query22	2.63	1.75	1.68
query23	17.03	1.08	0.83
query24	2.60	1.62	0.27
query25	0.23	0.20	0.26
query26	0.30	0.14	0.14
query27	0.05	0.05	0.05
query28	11.31	1.10	1.06
query29	12.59	3.24	3.27
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.24	0.48	0.47
query33	3.10	3.16	3.13
query34	17.09	4.50	4.45
query35	4.55	4.56	4.51
query36	0.67	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 107.29 s
Total hot run time: 32.06 s

github-actions · 2024-12-23T09:42:17Z

PR approved by at least one committer and no changes requested.

924060929 · 2024-12-24T07:26:10Z

run buildall

doris-robot · 2024-12-24T07:45:22Z

TPC-H: Total hot run time: 39820 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false

------ Round 1 ----------------------------------
q1	17691	7442	7282	7282
q2	2051	179	165	165
q3	10619	1118	1104	1104
q4	10220	715	725	715
q5	7606	2683	2659	2659
q6	237	151	146	146
q7	997	620	598	598
q8	9259	1850	1951	1850
q9	6608	6485	6456	6456
q10	7033	2289	2343	2289
q11	466	273	271	271
q12	429	218	223	218
q13	17780	2933	2958	2933
q14	241	209	210	209
q15	557	500	501	500
q16	659	576	566	566
q17	973	590	543	543
q18	7296	6748	6741	6741
q19	1336	1033	872	872
q20	453	181	188	181
q21	3963	3292	3207	3207
q22	369	315	317	315
Total cold run time: 106843 ms
Total hot run time: 39820 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7297	7210	7627	7210
q2	327	230	231	230
q3	2875	2770	3033	2770
q4	2109	1891	1809	1809
q5	5526	5573	5591	5573
q6	226	142	140	140
q7	2194	1770	1759	1759
q8	3377	3551	3481	3481
q9	8900	8896	8881	8881
q10	3681	3571	3500	3500
q11	604	500	497	497
q12	810	586	584	584
q13	12449	3086	2968	2968
q14	298	269	296	269
q15	555	491	518	491
q16	698	648	659	648
q17	1802	1567	1557	1557
q18	7811	7599	7672	7599
q19	1659	1439	1492	1439
q20	2069	1812	1824	1812
q21	5371	5236	5274	5236
q22	620	606	584	584
Total cold run time: 71258 ms
Total hot run time: 59037 ms

doris-robot · 2024-12-24T07:57:31Z

TPC-DS: Total hot run time: 190328 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false

query1	975	390	372	372
query2	6517	2353	2357	2353
query3	6716	215	215	215
query4	33880	23543	23966	23543
query5	4293	459	472	459
query6	293	192	192	192
query7	4629	321	325	321
query8	308	244	238	238
query9	9426	2782	2771	2771
query10	478	267	250	250
query11	18338	15150	15123	15123
query12	159	107	111	107
query13	1731	423	435	423
query14	10955	6722	6973	6722
query15	294	178	182	178
query16	8278	460	472	460
query17	1738	586	575	575
query18	2158	314	302	302
query19	389	150	150	150
query20	113	101	109	101
query21	205	99	105	99
query22	4651	4380	4542	4380
query23	34264	33056	33235	33056
query24	11430	2451	2429	2429
query25	674	376	406	376
query26	1755	156	153	153
query27	2951	335	332	332
query28	8146	2467	2493	2467
query29	1027	420	415	415
query30	307	148	160	148
query31	1014	810	784	784
query32	95	59	60	59
query33	777	314	285	285
query34	967	514	503	503
query35	950	725	730	725
query36	1075	944	960	944
query37	291	77	79	77
query38	4164	4356	4033	4033
query39	1447	1456	1427	1427
query40	281	102	103	102
query41	48	47	47	47
query42	116	101	101	101
query43	524	498	492	492
query44	1251	815	839	815
query45	183	158	164	158
query46	1142	716	714	714
query47	1937	1838	1855	1838
query48	403	330	313	313
query49	1298	410	385	385
query50	810	382	384	382
query51	7221	7119	7022	7022
query52	100	96	88	88
query53	257	185	185	185
query54	1174	410	412	410
query55	78	78	83	78
query56	254	251	237	237
query57	1243	1128	1094	1094
query58	232	236	243	236
query59	3244	3043	2967	2967
query60	285	244	249	244
query61	113	149	109	109
query62	933	728	738	728
query63	219	185	188	185
query64	5077	668	705	668
query65	3237	3180	3216	3180
query66	1415	324	302	302
query67	15817	15526	15601	15526
query68	5869	578	556	556
query69	420	253	248	248
query70	1186	1164	1149	1149
query71	450	279	243	243
query72	6552	4155	4007	4007
query73	787	361	363	361
query74	10415	9116	9127	9116
query75	3470	2684	2692	2684
query76	3561	1069	1084	1069
query77	554	283	282	282
query78	10420	9524	9416	9416
query79	2430	609	613	609
query80	1094	419	493	419
query81	532	232	225	225
query82	944	119	122	119
query83	254	155	144	144
query84	243	73	68	68
query85	1348	309	312	309
query86	442	302	303	302
query87	4446	4268	4343	4268
query88	4064	2228	2275	2228
query89	419	295	303	295
query90	2134	188	190	188
query91	142	111	109	109
query92	63	52	52	52
query93	1758	563	563	563
query94	1124	285	301	285
query95	354	254	263	254
query96	609	279	285	279
query97	2856	2692	2685	2685
query98	226	201	202	201
query99	1677	1442	1459	1442
Total cold run time: 307593 ms
Total hot run time: 190328 ms

doris-robot · 2024-12-24T08:03:01Z

ClickBench: Total hot run time: 32.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.44	0.43	0.40
query6	1.17	0.65	0.64
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.51
query10	0.55	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.61
query14	2.76	2.77	2.72
query15	0.89	0.82	0.82
query16	0.37	0.39	0.36
query17	1.04	1.04	1.05
query18	0.23	0.22	0.21
query19	1.98	1.88	1.97
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.81	1.86	2.42
query23	16.93	0.93	0.77
query24	3.43	0.75	1.67
query25	0.23	0.14	0.22
query26	0.40	0.14	0.14
query27	0.06	0.04	0.04
query28	10.53	1.09	1.06
query29	12.58	3.30	3.26
query30	0.25	0.06	0.06
query31	2.85	0.39	0.39
query32	3.24	0.46	0.44
query33	3.10	3.08	3.15
query34	17.15	4.51	4.49
query35	4.50	4.49	4.48
query36	0.68	0.48	0.50
query37	0.09	0.06	0.05
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.14
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.51 s
Total hot run time: 32.55 s

use binary search to speed up prune partitions. test: datetime partition column which range from 2020-07-02 to 2024-11-13, one date for one partition, query one partition for example `dt = '2024-01-02'`, this pr can speed up from QPS 160 to QPS 610

… (#51966) Cherry-picked from #44586 Co-authored-by: 924060929 <lanhuajian@selectdb.com>

Followup #44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**

Followup apache#44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**

Followup #44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**

Followup apache#44586 Enable binary search partition pruning optimization for Hive external tables. This PR adds binary search partition pruning support for Hive tables by: - Adding `getSortedPartitionRanges()` method to `ExternalTable` base class - Maintaining sorted partition ranges directly in `HivePartitionValues` for cache lifecycle consistency - Overriding `getSortedPartitionRanges()` in `HMSExternalTable` to provide sorted ranges **Performance improvement (20000 partitions, 1000 queries):** - Binary search enabled: **4.548 seconds** - Binary search disabled: **12.849 seconds** - **~2.8x faster**

924060929 requested review from CalvinKirs, dataroaring and morningman as code owners November 26, 2024 02:27

924060929 force-pushed the sort_partitions branch from 98f7178 to 5db0287 Compare November 26, 2024 02:53

morrySnow reviewed Nov 28, 2024

View reviewed changes

feiniaofeiafei reviewed Dec 3, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java Show resolved Hide resolved

feiniaofeiafei reviewed Dec 3, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java Show resolved Hide resolved

924060929 force-pushed the sort_partitions branch from 5db0287 to 3674062 Compare December 17, 2024 12:58

924060929 force-pushed the sort_partitions branch from 3674062 to 5bd7802 Compare December 18, 2024 05:13

924060929 force-pushed the sort_partitions branch from bafd5fb to 3b878fc Compare December 20, 2024 07:01

924060929 force-pushed the sort_partitions branch from 3b878fc to 264f436 Compare December 20, 2024 07:05

924060929 force-pushed the sort_partitions branch from 264f436 to 197aa64 Compare December 20, 2024 14:30

morrySnow approved these changes Dec 23, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 23, 2024

feiniaofeiafei approved these changes Dec 23, 2024

View reviewed changes

morningman approved these changes Dec 23, 2024

View reviewed changes

924060929 merged commit 8e09e1b into apache:master Dec 24, 2024
19 checks passed

morrySnow mentioned this pull request Dec 27, 2024

[fix](nereids)fix rangeSet intersect in PartitionPredicateToRange #46087

Merged

16 tasks

924060929 deleted the sort_partitions branch January 10, 2025 07:18

morrySnow added the dev/3.1.x label Feb 11, 2025

morrySnow added dev/3.1.x and removed dev/3.1.x labels Jun 19, 2025

github-actions bot mentioned this pull request Jun 19, 2025

branch-3.1: [opt](nereids) use binary search to prune partitions #44586 #51966

Merged

morrySnow pushed a commit that referenced this pull request Jun 20, 2025

branch-3.1: [opt](nereids) use binary search to prune partitions #44586…

3b868c9

… (#51966) Cherry-picked from #44586 Co-authored-by: 924060929 <lanhuajian@selectdb.com>

morrySnow added dev/3.1.0-merged and removed dev/3.1.x labels Jun 20, 2025

zy-kkk mentioned this pull request Dec 9, 2025

[opt](hive) use binary search to prune hive partitions #58877

Merged

Comments

Conversation

924060929 commented Nov 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 26, 2024

Uh oh!

924060929 commented Nov 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

924060929 commented Dec 17, 2024

Uh oh!

924060929 commented Dec 18, 2024

Uh oh!

github-actions bot commented Dec 18, 2024

sh-checker report

Uh oh!

924060929 commented Dec 18, 2024

Uh oh!

github-actions bot commented Dec 18, 2024

sh-checker report

Uh oh!

924060929 commented Dec 18, 2024

Uh oh!

github-actions bot commented Dec 18, 2024

sh-checker report

Uh oh!

924060929 commented Dec 18, 2024

Uh oh!

github-actions bot commented Dec 18, 2024

sh-checker report

Uh oh!

924060929 commented Dec 20, 2024

Uh oh!

924060929 commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

924060929 commented Dec 20, 2024

Uh oh!

doris-robot commented Dec 20, 2024

Uh oh!

924060929 commented Dec 23, 2024

Uh oh!

doris-robot commented Dec 23, 2024

Uh oh!

doris-robot commented Dec 23, 2024

Uh oh!

doris-robot commented Dec 23, 2024

Uh oh!

github-actions bot commented Dec 23, 2024

Uh oh!

924060929 commented Dec 24, 2024

Uh oh!

doris-robot commented Dec 24, 2024

Uh oh!

doris-robot commented Dec 24, 2024

Uh oh!

doris-robot commented Dec 24, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

924060929 commented Nov 26, 2024 •

edited

Loading

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`