Skip to content

[fix](column predicate)fix topn&bloom&bitmap filter convert to column predicate error.#60262

Merged
morningman merged 3 commits intoapache:masterfrom
hubgeter:fix_column_pred
Jan 29, 2026
Merged

[fix](column predicate)fix topn&bloom&bitmap filter convert to column predicate error.#60262
morningman merged 3 commits intoapache:masterfrom
hubgeter:fix_column_pred

Conversation

@hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Jan 27, 2026

What problem does this PR solve?

Problem Summary:

  1. fix [fix](parquet)fix parquet reader cannot push down conjuncts for min-max filter #60197 :
    Null pointer exceptions may occur when processing non-slot topn values.
*** Query id: 45488371394ee0-aac81c6593b2f02a ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1769444368 (unix time) try "date -d @1769444368" if you are using GNU date ***
*** Current BE git commitID: a0eb8c0c71 ***
*** SIGSEGV address not mapped to object (@0x18) received by PID 13103 (TID 13838 OR 0x7ff0423ee640) from PID 24; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/common/signal_handler.h:420
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007FF250EE5520 in /lib/x86_64-linux-gnu/libc.so.6
 4# doris::vectorized::RuntimePredicate::update(doris::vectorized::Field const&) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/runtime_predicate.cpp:219
 5# doris::pipeline::SortSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/sort_sink_operator.cpp:152
 6# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/pipeline_task.cpp:598
  1. fix : file scan node push down bloom_filter & bitmap_filter for min-max filter.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 27, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@hubgeter hubgeter changed the title [fix](topn)fix topn&rf expr convert to column predicate error. [fix](topn)fix topn&bloom&bitmap filter convert to column predicate error. Jan 27, 2026
@hubgeter hubgeter changed the title [fix](topn)fix topn&bloom&bitmap filter convert to column predicate error. [fix](column predicate)fix topn&bloom&bitmap filter convert to column predicate error. Jan 27, 2026
BiteTheDDDDt
BiteTheDDDDt previously approved these changes Jan 27, 2026
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 27, 2026
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 32843 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 64fa5e1181eedfd5003a5f03ce9f7cfbc01c9513, data reload: false

------ Round 1 ----------------------------------
q1	17628	5340	5090	5090
q2	2100	317	189	189
q3	10181	1342	771	771
q4	10218	850	317	317
q5	7508	2199	1898	1898
q6	200	183	148	148
q7	916	730	594	594
q8	9266	1431	1139	1139
q9	5127	4878	4845	4845
q10	6804	1942	1585	1585
q11	513	293	281	281
q12	331	381	230	230
q13	17774	4073	3250	3250
q14	242	243	215	215
q15	903	826	813	813
q16	693	710	629	629
q17	643	823	470	470
q18	6836	6419	7661	6419
q19	1254	1035	636	636
q20	405	366	264	264
q21	3015	2243	2066	2066
q22	1128	1110	994	994
Total cold run time: 103685 ms
Total hot run time: 32843 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5586	5792	5583	5583
q2	258	351	247	247
q3	2478	2842	2430	2430
q4	1519	1959	1465	1465
q5	4580	4669	4649	4649
q6	235	186	139	139
q7	2098	1943	1806	1806
q8	2574	2678	2432	2432
q9	7424	7458	7576	7458
q10	2835	3172	2643	2643
q11	558	473	454	454
q12	774	781	616	616
q13	3784	4356	3247	3247
q14	269	275	257	257
q15	835	795	790	790
q16	642	680	637	637
q17	1070	1196	1285	1196
q18	7698	7307	7382	7307
q19	818	782	789	782
q20	1961	2034	1918	1918
q21	4616	4166	4055	4055
q22	1061	1061	951	951
Total cold run time: 53673 ms
Total hot run time: 51062 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 64fa5e1181eedfd5003a5f03ce9f7cfbc01c9513, data reload: false

query1	0.05	0.04	0.05
query2	0.10	0.05	0.04
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.25
query6	1.17	0.67	0.67
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.57	0.56	0.55
query11	0.14	0.09	0.09
query12	0.14	0.10	0.10
query13	0.64	0.62	0.63
query14	1.06	1.04	1.05
query15	0.89	0.86	0.87
query16	0.40	0.39	0.38
query17	1.12	1.15	1.09
query18	0.23	0.21	0.21
query19	2.03	2.03	2.00
query20	0.03	0.02	0.02
query21	15.41	0.27	0.15
query22	5.20	0.06	0.04
query23	15.95	0.27	0.10
query24	1.82	0.47	0.80
query25	0.12	0.12	0.07
query26	0.14	0.13	0.13
query27	0.06	0.06	0.04
query28	4.92	1.14	0.96
query29	12.55	3.98	3.16
query30	0.28	0.13	0.11
query31	2.81	0.63	0.41
query32	3.23	0.59	0.51
query33	3.29	3.21	3.26
query34	16.01	5.41	4.70
query35	4.77	4.80	4.77
query36	0.66	0.51	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.02
query40	0.19	0.17	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 99.17 s
Total hot run time: 28.39 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.49% (19160/36501)
Line Coverage 35.90% (178248/496537)
Region Coverage 32.37% (137861/425947)
Branch Coverage 33.30% (59667/179197)

yiguolei pushed a commit that referenced this pull request Jan 27, 2026
…t to column predicate error. (#60262) (#60268)

### What problem does this PR solve?
Problem Summary:
bp #60262
@hubgeter
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 28, 2026
@doris-robot
Copy link

TPC-H: Total hot run time: 31935 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7a1f6ef34867d93e44eeec4a36450bd976880f58, data reload: false

------ Round 1 ----------------------------------
q1	17609	5232	5088	5088
q2	2022	307	191	191
q3	10277	1297	738	738
q4	10200	818	320	320
q5	7511	2126	1893	1893
q6	194	179	148	148
q7	910	741	603	603
q8	9272	1359	1065	1065
q9	5242	4917	4891	4891
q10	6806	1968	1576	1576
q11	501	295	261	261
q12	332	386	220	220
q13	17774	4032	3201	3201
q14	229	233	221	221
q15	913	828	802	802
q16	670	675	623	623
q17	629	800	456	456
q18	6836	6585	6588	6585
q19	1243	985	624	624
q20	393	341	223	223
q21	2588	1964	1934	1934
q22	354	310	272	272
Total cold run time: 102505 ms
Total hot run time: 31935 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5342	5300	5321	5300
q2	269	336	250	250
q3	2155	2674	2292	2292
q4	1349	1719	1299	1299
q5	4262	4211	4272	4211
q6	215	187	140	140
q7	2114	2114	1892	1892
q8	2566	2524	2396	2396
q9	7517	7502	7518	7502
q10	2878	3120	2604	2604
q11	552	472	455	455
q12	662	750	640	640
q13	3819	4619	3466	3466
q14	298	331	298	298
q15	898	811	832	811
q16	700	750	676	676
q17	1155	1381	1365	1365
q18	8581	7935	7956	7935
q19	840	821	866	821
q20	2057	2138	1971	1971
q21	4844	4348	4048	4048
q22	631	549	531	531
Total cold run time: 53704 ms
Total hot run time: 50903 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7a1f6ef34867d93e44eeec4a36450bd976880f58, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.05	0.05
query3	0.25	0.09	0.08
query4	1.60	0.11	0.11
query5	0.26	0.26	0.26
query6	1.16	0.68	0.68
query7	0.03	0.03	0.02
query8	0.05	0.03	0.03
query9	0.55	0.50	0.49
query10	0.56	0.55	0.54
query11	0.14	0.10	0.10
query12	0.13	0.10	0.11
query13	0.64	0.62	0.61
query14	1.08	1.06	1.07
query15	0.86	0.86	0.87
query16	0.39	0.38	0.38
query17	1.07	1.12	1.08
query18	0.22	0.20	0.20
query19	2.07	2.06	2.00
query20	0.02	0.02	0.01
query21	15.39	0.27	0.15
query22	5.25	0.05	0.04
query23	15.99	0.28	0.11
query24	0.95	0.23	0.51
query25	0.09	0.08	0.06
query26	0.16	0.13	0.13
query27	0.06	0.11	0.06
query28	3.89	1.16	0.96
query29	12.54	4.01	3.21
query30	0.28	0.13	0.12
query31	2.82	0.63	0.40
query32	3.24	0.59	0.51
query33	3.16	3.37	3.24
query34	16.13	5.43	4.72
query35	4.82	4.78	4.82
query36	0.64	0.49	0.50
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.21 s
Total hot run time: 28.29 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.72% (19255/36526)
Line Coverage 36.09% (178864/495666)
Region Coverage 32.54% (138747/426354)
Branch Coverage 33.48% (60041/179343)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 20.00% (1/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 69.84% (25007/35804)
Line Coverage 52.29% (258518/494433)
Region Coverage 49.80% (214487/430672)
Branch Coverage 51.00% (91831/180059)

1 similar comment
@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 20.00% (1/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 69.84% (25007/35804)
Line Coverage 52.29% (258518/494433)
Region Coverage 49.80% (214487/430672)
Branch Coverage 51.00% (91831/180059)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2026
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 22ce979 into apache:master Jan 29, 2026
31 of 32 checks passed
morningman pushed a commit that referenced this pull request Jan 29, 2026
… predicate error. (#60262)

### What problem does this PR solve?
Problem Summary:
1. fix #60197  :  
Null pointer exceptions may occur when processing non-slot topn values.
```
*** Query id: 45488371394ee0-aac81c6593b2f02a ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1769444368 (unix time) try "date -d @1769444368" if you are using GNU date ***
*** Current BE git commitID: a0eb8c0 ***
*** SIGSEGV address not mapped to object (@0x18) received by PID 13103 (TID 13838 OR 0x7ff0423ee640) from PID 24; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/common/signal_handler.h:420
 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
 3# 0x00007FF250EE5520 in /lib/x86_64-linux-gnu/libc.so.6
 4# doris::vectorized::RuntimePredicate::update(doris::vectorized::Field const&) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/runtime_predicate.cpp:219
 5# doris::pipeline::SortSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/sort_sink_operator.cpp:152
 6# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/pipeline_task.cpp:598
 ````
 
 2. fix : file scan node push down bloom_filter & bitmap_filter for min-max filter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants

Comments