[opt](inverted index) Add built_in_analyzer parser alias and modify custom analyzer logic#57055

zzzxl1993 · 2025-10-16T08:28:12Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Add built-in parser alias and support same-name config for custom analyzer components

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-10-16T08:28:18Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zzzxl1993 · 2025-10-16T08:28:35Z

run buildall

hello-stephen · 2025-10-16T09:39:29Z

FE UT Coverage Report

Increment line coverage 15.00% (6/40) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-10-16T11:36:34Z

BE UT Coverage Report

Increment line coverage 91.18% (124/136) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.60% (17849/33936)
Line Coverage	37.75% (161965/429004)
Region Coverage	32.18% (123356/383284)
Branch Coverage	33.60% (54206/161312)

be/src/olap/inverted_index_parser.h

csun5285

LGTM

github-actions · 2025-10-17T02:31:19Z

PR approved by anyone and no changes requested.

zzzxl1993 · 2025-10-17T05:18:13Z

run buildall

hello-stephen · 2025-10-17T06:29:02Z

FE UT Coverage Report

Increment line coverage 15.00% (6/40) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-10-17T07:06:23Z

ClickBench: Total hot run time: 32.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 74612b2c19b7326d7b159ecc8cfeaa3403c9dbd8, data reload: false

query1	0.06	0.06	0.06
query2	0.10	0.06	0.06
query3	0.26	0.10	0.11
query4	1.62	0.12	0.12
query5	0.30	0.27	0.29
query6	1.18	0.69	0.68
query7	0.04	0.03	0.03
query8	0.08	0.05	0.05
query9	0.67	0.58	0.60
query10	0.64	0.64	0.63
query11	0.20	0.13	0.13
query12	0.18	0.14	0.13
query13	0.64	0.63	0.64
query14	1.08	1.08	1.06
query15	0.91	0.90	0.97
query16	0.44	0.43	0.43
query17	1.14	1.25	1.17
query18	0.24	0.22	0.23
query19	2.10	1.91	2.01
query20	0.01	0.02	0.02
query21	15.37	1.05	0.68
query22	0.82	1.25	0.70
query23	14.86	1.60	0.69
query24	7.26	1.68	0.84
query25	0.45	0.16	0.10
query26	0.62	0.20	0.16
query27	0.07	0.06	0.07
query28	9.27	1.46	0.95
query29	12.61	4.72	3.84
query30	0.29	0.15	0.12
query31	2.85	0.67	0.43
query32	3.26	0.58	0.49
query33	3.21	3.15	3.16
query34	16.07	5.58	4.90
query35	5.01	4.98	4.96
query36	0.74	0.54	0.52
query37	0.12	0.09	0.08
query38	0.08	0.05	0.05
query39	0.04	0.04	0.04
query40	0.20	0.16	0.16
query41	0.10	0.03	0.04
query42	0.04	0.03	0.03
query43	0.06	0.04	0.04
Total cold run time: 105.29 s
Total hot run time: 32.2 s

doris-robot · 2025-10-17T08:34:46Z

BE UT Coverage Report

Increment line coverage 91.18% (124/136) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.60% (17857/33948)
Line Coverage	37.76% (162041/429139)
Region Coverage	32.25% (123696/383583)
Branch Coverage	33.62% (54248/161360)

airborne12

LGTM

…lyzer logic

airborne12 · 2025-10-17T12:45:19Z

run buildall

hello-stephen · 2025-10-17T13:53:09Z

FE UT Coverage Report

Increment line coverage 15.00% (6/40) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-10-17T15:14:45Z

ClickBench: Total hot run time: 30.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dcb05c4894dd2f6bcddb671c740e46c382b40265, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.12	0.12
query5	0.29	0.28	0.26
query6	1.17	0.66	0.67
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.64	0.54	0.52
query10	0.59	0.59	0.60
query11	0.17	0.13	0.12
query12	0.18	0.13	0.13
query13	0.65	0.63	0.63
query14	1.06	1.05	1.04
query15	0.90	0.88	0.88
query16	0.41	0.41	0.41
query17	1.08	1.08	1.11
query18	0.23	0.21	0.20
query19	2.01	1.91	1.88
query20	0.02	0.02	0.01
query21	15.40	0.97	0.60
query22	0.77	1.26	0.76
query23	14.75	1.40	0.65
query24	7.00	1.63	0.49
query25	0.49	0.27	0.08
query26	0.56	0.17	0.15
query27	0.07	0.06	0.07
query28	10.08	1.48	0.94
query29	12.56	4.08	3.34
query30	0.28	0.14	0.11
query31	2.83	0.63	0.40
query32	3.24	0.57	0.52
query33	3.17	3.11	3.20
query34	16.04	5.58	4.95
query35	4.99	4.94	4.96
query36	0.71	0.53	0.52
query37	0.10	0.08	0.08
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.99 s
Total hot run time: 30.8 s

doris-robot · 2025-10-17T15:48:09Z

BE UT Coverage Report

Increment line coverage 91.18% (124/136) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.60% (17869/33972)
Line Coverage	37.82% (162059/428507)
Region Coverage	32.24% (123572/383340)
Branch Coverage	33.65% (54155/160953)

hello-stephen · 2025-10-17T17:05:05Z

BE Regression && UT Coverage Report

Increment line coverage 93.85% (122/130) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.33% (23758/33306)
Line Coverage	57.73% (247230/428260)
Region Coverage	52.80% (205012/388299)
Branch Coverage	54.59% (88350/161848)

hello-stephen · 2025-10-17T17:17:01Z

FE Regression Coverage Report

Increment line coverage 35.00% (14/40) 🎉
Increment coverage report
Complete coverage report

airborne12

LGTM

github-actions · 2025-10-20T02:41:43Z

PR approved by at least one committer and no changes requested.

…ustom analyzer logic (apache#57055) Add built-in parser alias and support same-name config for custom analyzer components

…asic and icu tokenizer (#57266) #56243 #57055

…ar_filter, basic and icu tokenizer #56243 #57055 (#57137) picked from #56243 #57055

airborne12 reviewed Oct 17, 2025

View reviewed changes

be/src/olap/inverted_index_parser.h Outdated Show resolved Hide resolved

csun5285 approved these changes Oct 17, 2025

View reviewed changes

github-actions bot added the reviewed label Oct 17, 2025

zzzxl1993 changed the title ~~[opt](inverted index) Add built-in parser alias and modify custom analyzer logic~~ [opt](inverted index) Add built_in_analyzer parser alias and modify custom analyzer logic Oct 17, 2025

airborne12 added the dev/3.1.x label Oct 17, 2025

airborne12 approved these changes Oct 17, 2025

View reviewed changes

zzzxl1993 added 2 commits October 17, 2025 20:44

[opt](inverted index) Add built-in parser alias and modify custom ana…

4fc0999

…lyzer logic

fix

dcb05c4

airborne12 force-pushed the 202510161623 branch from 74612b2 to dcb05c4 Compare October 17, 2025 12:44

airborne12 approved these changes Oct 20, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2025

airborne12 merged commit 5c610cd into apache:master Oct 20, 2025
25 of 26 checks passed

github-actions bot added the dev/3.1.x-conflict label Oct 20, 2025

zzzxl1993 mentioned this pull request Oct 20, 2025

branch-3.1: [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243 #57055 #57137

Merged

airborne12 added the dev/4.0.x label Oct 23, 2025

github-actions bot added the dev/4.0.x-conflict label Oct 23, 2025

zzzxl1993 mentioned this pull request Oct 23, 2025

[opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #57266

Merged

yiguolei pushed a commit that referenced this pull request Oct 24, 2025

[opt](inverted index) add custom analyzer support with char_filter, b…

d498425

…asic and icu tokenizer (#57266) #56243 #57055

yiguolei added dev/4.0.1-merged and removed dev/4.0.x dev/4.0.x-conflict labels Oct 24, 2025

morrySnow pushed a commit that referenced this pull request Oct 30, 2025

branch-3.1: [opt](inverted index) add custom analyzer support with ch…

347775b

…ar_filter, basic and icu tokenizer #56243 #57055 (#57137) picked from #56243 #57055

morrySnow added dev/3.1.3-merged and removed dev/3.1.x dev/3.1.x-conflict labels Oct 30, 2025

yiguolei mentioned this pull request Nov 5, 2025

4.0.1 Release Notes #57717

Closed

Conversation

zzzxl1993 commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Oct 16, 2025

Uh oh!

zzzxl1993 commented Oct 16, 2025

Uh oh!

hello-stephen commented Oct 16, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Oct 16, 2025

BE UT Coverage Report

Uh oh!

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 17, 2025

Uh oh!

zzzxl1993 commented Oct 17, 2025

Uh oh!

hello-stephen commented Oct 17, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Oct 17, 2025

Uh oh!

doris-robot commented Oct 17, 2025

BE UT Coverage Report

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

airborne12 commented Oct 17, 2025

Uh oh!

hello-stephen commented Oct 17, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Oct 17, 2025

Uh oh!

doris-robot commented Oct 17, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 17, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Oct 17, 2025

FE Regression Coverage Report

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Comments

zzzxl1993 commented Oct 16, 2025 •

edited

Loading