Skip to content

[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform#43391

Merged
morrySnow merged 34 commits intoapache:masterfrom
feiniaofeiafei:project_filter_transform_const
Dec 2, 2024
Merged

[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform#43391
morrySnow merged 34 commits intoapache:masterfrom
feiniaofeiafei:project_filter_transform_const

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Nov 7, 2024 •
edited
Loading

What problem does this PR solve?

This PR introduces two main changes:

  1. Adds an optional constant value to the uniform attribute in DataTrait. A slot with a constant value that is not null will be considered uniform and not null.
  2. Introduces a new transform rule: EliminateGroupByKeyByUniform, which utilizes the newly added part of the uniform attribute. Following is example transformation:
    +--aggregate(group by a,b output a,b,max(c))
    (a is uniform and not null: e.g. a is projection 2 as a in logicalProject)
    ->
    +--aggregate(group by b output b,any_value(a) as a,max(c))

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 7, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 7, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morrySnow
morrySnow reviewed Nov 7, 2024
View reviewed changes
...ain/java/org/apache/doris/nereids/rules/rewrite/PredicateDrivenProjectionSimplification.java Outdated
import java.util.Map;
import java.util.Set;

/**ProjectFilterTransform*/
Copy link
Contributor

@morrySnow morrySnow Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

class comment should contain what the rule want to do and how

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 7, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 7, 2024

TPC-H: Total hot run time: 41389 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8bd01425901bc1de525e568ff0c226f281889a31, data reload: false

------ Round 1 ----------------------------------
q1	17572	7461	7337	7337
q2	2060	177	184	177
q3	10660	1082	1143	1082
q4	10562	879	824	824
q5	7758	3071	3071	3071
q6	247	144	145	144
q7	1022	602	611	602
q8	9373	1971	2024	1971
q9	6583	6458	6555	6458
q10	7115	2460	2429	2429
q11	483	250	262	250
q12	404	210	212	210
q13	17791	2993	3005	2993
q14	241	216	212	212
q15	587	526	517	517
q16	673	581	600	581
q17	979	528	557	528
q18	7272	6753	6650	6650
q19	1338	1025	942	942
q20	467	182	176	176
q21	3968	3302	3246	3246
q22	1119	989	1005	989
Total cold run time: 108274 ms
Total hot run time: 41389 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7336	7257	7293	7257
q2	346	247	258	247
q3	2996	2989	2925	2925
q4	2141	1903	1841	1841
q5	5760	5829	5849	5829
q6	227	143	142	142
q7	2286	1856	1841	1841
q8	3447	3561	3556	3556
q9	8912	8918	8966	8918
q10	3612	3637	3582	3582
q11	649	510	515	510
q12	851	627	650	627
q13	10088	3218	3167	3167
q14	322	270	298	270
q15	601	579	562	562
q16	674	656	648	648
q17	1859	1667	1679	1667
q18	8188	7827	7627	7627
q19	1714	1587	1579	1579
q20	2092	1899	1901	1899
q21	5586	5514	5440	5440
q22	1154	1052	1047	1047
Total cold run time: 70841 ms
Total hot run time: 61181 ms

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 7, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 7, 2024

TPC-H: Total hot run time: 41292 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e668096307fe4a2ded45eb474b160a5d2f76e293, data reload: false

------ Round 1 ----------------------------------
q1	17613	7399	7282	7282
q2	2069	186	191	186
q3	10529	1055	1165	1055
q4	10498	905	880	880
q5	7755	3087	3006	3006
q6	239	145	149	145
q7	1028	630	617	617
q8	9380	2020	2087	2020
q9	6559	6410	6417	6410
q10	7101	2406	2413	2406
q11	459	252	253	252
q12	397	212	209	209
q13	17793	3006	3027	3006
q14	237	210	205	205
q15	577	508	506	506
q16	652	570	588	570
q17	966	544	504	504
q18	7337	6755	6698	6698
q19	1332	1066	1014	1014
q20	450	180	179	179
q21	3964	3291	3182	3182
q22	1120	1014	960	960
Total cold run time: 108055 ms
Total hot run time: 41292 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7300	7283	7206	7206
q2	351	257	249	249
q3	2974	2966	2995	2966
q4	2067	1948	1811	1811
q5	5756	5801	5840	5801
q6	232	137	139	137
q7	2300	1846	1850	1846
q8	3418	3524	3531	3524
q9	8936	8919	8888	8888
q10	3611	3561	3559	3559
q11	614	500	498	498
q12	848	651	627	627
q13	10752	3189	3254	3189
q14	297	273	268	268
q15	607	568	548	548
q16	698	641	650	641
q17	1855	1644	1613	1613
q18	8201	7694	7757	7694
q19	1679	1505	1511	1505
q20	2171	1885	1873	1873
q21	5723	5430	5600	5430
q22	1161	1030	1053	1030
Total cold run time: 71551 ms
Total hot run time: 60903 ms

@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch from e668096 to 452bb5f Compare November 12, 2024 13:37
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 12, 2024

run buildall

@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch from 0c01e45 to 6333e6f Compare November 12, 2024 13:40
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 12, 2024

run buildall

@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch 2 times, most recently from 6f13f10 to 892b7ec Compare November 13, 2024 04:22
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 13, 2024

run buildall

@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch 2 times, most recently from 6b4a551 to 852065c Compare November 14, 2024 09:46
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 14, 2024

run buidlall

@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch 2 times, most recently from b393ff3 to 0c470a6 Compare November 14, 2024 10:04
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 14, 2024

run buildall

@feiniaofeiafei feiniaofeiafei changed the title [feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform and PredicateDrivenProjectionSimplification [feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform Nov 14, 2024
@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch 2 times, most recently from 73c21af to c2bc25a Compare November 15, 2024 10:39
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 15, 2024

run buildall

@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch 2 times, most recently from cdb345c to c99e369 Compare November 15, 2024 10:47
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 15, 2024

run buildall

@feiniaofeiafei feiniaofeiafei force-pushed the project_filter_transform_const branch 2 times, most recently from 1d2ad46 to d211163 Compare November 15, 2024 12:01
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 27, 2024

ClickBench: Total hot run time: 32.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c2bb29dd64f003618525558e10c77773c15650e, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.41	0.41
query6	1.16	0.67	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.53	0.52
query10	0.55	0.55	0.56
query11	0.15	0.11	0.10
query12	0.14	0.12	0.10
query13	0.61	0.61	0.59
query14	2.71	2.73	2.84
query15	0.92	0.83	0.83
query16	0.39	0.39	0.39
query17	1.05	1.07	1.03
query18	0.22	0.21	0.21
query19	1.98	1.84	2.09
query20	0.02	0.02	0.01
query21	15.36	0.61	0.60
query22	2.56	2.56	1.37
query23	17.18	0.88	0.71
query24	3.33	1.79	0.85
query25	0.20	0.15	0.12
query26	0.57	0.14	0.14
query27	0.05	0.05	0.04
query28	9.83	1.12	1.09
query29	12.58	3.28	3.23
query30	0.25	0.06	0.06
query31	2.85	0.39	0.39
query32	3.26	0.47	0.47
query33	3.04	3.00	3.05
query34	17.01	4.48	4.52
query35	4.55	4.58	4.56
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.65 s
Total hot run time: 32.14 s

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 27, 2024

run p0

xzj7019
xzj7019 reviewed Nov 28, 2024
View reviewed changes
...sion-test/suites/nereids_rules_p0/eliminate_gby_key/eliminate_group_by_key_by_uniform.groovy
sql "insert into test1 values(1,1),(2,1),(3,1);"
sql "create table test2(a int, b int) distributed by hash(a) properties('replication_num'='1');"
sql "insert into test2 values(1,105),(2,105);"
qt_full_join_uniform_should_not_eliminate_group_by_key "select t2.b,t1.b from test1 t1 full join (select * from test2 where b=105) t2 on t1.a=t2.a group by t2.b,t1.b order by 1,2;"
Copy link
Contributor

@xzj7019 xzj7019 Nov 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the original code have bug for this case?

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 29, 2024

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit ed4f7fb into apache:master Dec 2, 2024
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Dec 6, 2024
@feiniaofeiafei
[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform (apach…
0bcf46c 
…e#43391)

This PR introduces two main changes:
1. Adds an optional constant value to the uniform attribute in
DataTrait. A slot with a constant value that is not null will be
considered uniform and not null.
2. Introduces a new transform rule: EliminateGroupByKeyByUniform, which
utilizes the newly added part of the uniform attribute. Following is
example transformation:

from

 +--aggregate(group by a,b output a,b,max(c))
(a is uniform and not null: e.g. a is projection 2 as a in
logicalProject)

to

+--aggregate(group by b output b,any_value(a) as a,max(c))
@feiniaofeiafei feiniaofeiafei mentioned this pull request Dec 6, 2024
Merged
morrySnow pushed a commit that referenced this pull request Dec 10, 2024
@feiniaofeiafei
[feat](nereids) add rewrite rule :EliminateGroupByKeyByUniform (#43391)…
8a61eb9 
… (#45075)

cherry-pick #43391 to branch-3.0
@feiniaofeiafei feiniaofeiafei mentioned this pull request Jan 3, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Jan 8, 2025
@feiniaofeiafei
[fix](nereids) fix bug of LogicalRepeat computing uniform datatrait (#…
6de9d27 
…46352)

### What problem does this PR solve?

Related PR: #43391

Problem Summary:
repeat's uniform is not right, remove it temprarily
github-actions bot pushed a commit that referenced this pull request Jan 8, 2025
@feiniaofeiafei
[fix](nereids) fix bug of LogicalRepeat computing uniform datatrait (#…
b9dedfa 
…46352)

### What problem does this PR solve?

Related PR: #43391

Problem Summary:
repeat's uniform is not right, remove it temprarily
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Closed
@feiniaofeiafei feiniaofeiafei mentioned this pull request Sep 8, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Sep 10, 2025
@feiniaofeiafei
[enhance](nereids) EliminateGroupByKeyByUniform support rewrite in cte (
294b373 
#55680)

### What problem does this PR solve?

Related PR: #43391

Problem Summary:
EliminateGroupByKeyByUniform not support rewrite in cte in #43391, and
the restriction can be removed after
#52798, because #52798 implement the cteconsumer expression rewrite.
@seawinde seawinde mentioned this pull request Oct 14, 2025
Merged
16 tasks
dataroaring pushed a commit that referenced this pull request Nov 6, 2025
@seawinde
[fix](nereids) Fix not in aggregate's output err after eliminate by u…
a5e4e1c 
…niform when group sets exist (#56942)

Fix not in aggregate's output err after eliminate by uniform when group
sets exist

if query as following, would cause `ERROR 1105 (HY000): errCode = 2,
detailMessage = GROUPING_PREFIX_event_name_group not in aggregate's
output`

the pr fix this

```sql
SELECT
  CASE WHEN GROUPING(event_date) = 1 THEN '(TOTAL)' ELSE CAST(event_date AS VARCHAR) END AS event_date,
  user_id,
  MAX(conversion_level) AS conversion_level,
  CASE WHEN GROUPING(event_name_group) = 1 THEN '(TOTAL)' ELSE event_name_group END AS event_name_group
FROM
  (
    SELECT
      src.event_date,
      src.user_id,
      WINDOW_FUNNEL(
        3600 * 24 * 1,
        'default',
        src.event_time,
        src.event_name = 'shop_buy',
        src.event_name = 'shop_buy'
      ) AS conversion_level,
      src.event_name_group
    FROM
      (
        SELECT
          CAST(etb.`@dt` AS DATE) AS event_date,
          etb.`@event_name` AS event_name,
          etb.`@event_time` AS event_time,
          etb.`@event_name` AS event_name_group,
          etb.`@user_id` AS user_id
        FROM
          `test_event` AS etb
        WHERE
          etb.`@dt` between '2025-09-03 02:00:00' AND '2025-09-10 01:59:59'
          AND etb.`@event_name` = 'shop_buy'
          AND etb.`@user_id` IS NOT NULL
          AND etb.`@user_id` > '0'
      ) AS src
    GROUP BY
      src.event_date,
      src.user_id,
      src.event_name_group
  ) AS fwt
GROUP BY
  GROUPING SETS (
    (user_id),
    (user_id, event_date),
    (user_id, event_name_group),
    (user_id, event_date, event_name_group)
  );
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #43391 

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
github-actions bot pushed a commit that referenced this pull request Nov 6, 2025
@seawinde
[fix](nereids) Fix not in aggregate's output err after eliminate by u…
116cddb 
…niform when group sets exist (#56942)

Fix not in aggregate's output err after eliminate by uniform when group
sets exist

if query as following, would cause `ERROR 1105 (HY000): errCode = 2,
detailMessage = GROUPING_PREFIX_event_name_group not in aggregate's
output`

the pr fix this

```sql
SELECT
  CASE WHEN GROUPING(event_date) = 1 THEN '(TOTAL)' ELSE CAST(event_date AS VARCHAR) END AS event_date,
  user_id,
  MAX(conversion_level) AS conversion_level,
  CASE WHEN GROUPING(event_name_group) = 1 THEN '(TOTAL)' ELSE event_name_group END AS event_name_group
FROM
  (
    SELECT
      src.event_date,
      src.user_id,
      WINDOW_FUNNEL(
        3600 * 24 * 1,
        'default',
        src.event_time,
        src.event_name = 'shop_buy',
        src.event_name = 'shop_buy'
      ) AS conversion_level,
      src.event_name_group
    FROM
      (
        SELECT
          CAST(etb.`@dt` AS DATE) AS event_date,
          etb.`@event_name` AS event_name,
          etb.`@event_time` AS event_time,
          etb.`@event_name` AS event_name_group,
          etb.`@user_id` AS user_id
        FROM
          `test_event` AS etb
        WHERE
          etb.`@dt` between '2025-09-03 02:00:00' AND '2025-09-10 01:59:59'
          AND etb.`@event_name` = 'shop_buy'
          AND etb.`@user_id` IS NOT NULL
          AND etb.`@user_id` > '0'
      ) AS src
    GROUP BY
      src.event_date,
      src.user_id,
      src.event_name_group
  ) AS fwt
GROUP BY
  GROUPING SETS (
    (user_id),
    (user_id, event_date),
    (user_id, event_name_group),
    (user_id, event_date, event_name_group)
  );
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #43391 

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
yiguolei pushed a commit that referenced this pull request Nov 8, 2025
@seawinde @yiguolei
[fix](nereids) Fix not in aggregate's output err after eliminate by u…
d571b62 
…niform when group sets exist (#56942)

Fix not in aggregate's output err after eliminate by uniform when group
sets exist

if query as following, would cause `ERROR 1105 (HY000): errCode = 2,
detailMessage = GROUPING_PREFIX_event_name_group not in aggregate's
output`

the pr fix this

```sql
SELECT
  CASE WHEN GROUPING(event_date) = 1 THEN '(TOTAL)' ELSE CAST(event_date AS VARCHAR) END AS event_date,
  user_id,
  MAX(conversion_level) AS conversion_level,
  CASE WHEN GROUPING(event_name_group) = 1 THEN '(TOTAL)' ELSE event_name_group END AS event_name_group
FROM
  (
    SELECT
      src.event_date,
      src.user_id,
      WINDOW_FUNNEL(
        3600 * 24 * 1,
        'default',
        src.event_time,
        src.event_name = 'shop_buy',
        src.event_name = 'shop_buy'
      ) AS conversion_level,
      src.event_name_group
    FROM
      (
        SELECT
          CAST(etb.`@dt` AS DATE) AS event_date,
          etb.`@event_name` AS event_name,
          etb.`@event_time` AS event_time,
          etb.`@event_name` AS event_name_group,
          etb.`@user_id` AS user_id
        FROM
          `test_event` AS etb
        WHERE
          etb.`@dt` between '2025-09-03 02:00:00' AND '2025-09-10 01:59:59'
          AND etb.`@event_name` = 'shop_buy'
          AND etb.`@user_id` IS NOT NULL
          AND etb.`@user_id` > '0'
      ) AS src
    GROUP BY
      src.event_date,
      src.user_id,
      src.event_name_group
  ) AS fwt
GROUP BY
  GROUPING SETS (
    (user_id),
    (user_id, event_date),
    (user_id, event_name_group),
    (user_id, event_date, event_name_group)
  );
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #43391 

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
seawinde added a commit to seawinde/doris that referenced this pull request Nov 10, 2025
@seawinde
[fix](nereids) Fix not in aggregate's output err after eliminate by u…
4676966 
…niform when group sets exist (apache#56942)

Fix not in aggregate's output err after eliminate by uniform when group
sets exist

if query as following, would cause `ERROR 1105 (HY000): errCode = 2,
detailMessage = GROUPING_PREFIX_event_name_group not in aggregate's
output`

the pr fix this

```sql
SELECT
  CASE WHEN GROUPING(event_date) = 1 THEN '(TOTAL)' ELSE CAST(event_date AS VARCHAR) END AS event_date,
  user_id,
  MAX(conversion_level) AS conversion_level,
  CASE WHEN GROUPING(event_name_group) = 1 THEN '(TOTAL)' ELSE event_name_group END AS event_name_group
FROM
  (
    SELECT
      src.event_date,
      src.user_id,
      WINDOW_FUNNEL(
        3600 * 24 * 1,
        'default',
        src.event_time,
        src.event_name = 'shop_buy',
        src.event_name = 'shop_buy'
      ) AS conversion_level,
      src.event_name_group
    FROM
      (
        SELECT
          CAST(etb.`@dt` AS DATE) AS event_date,
          etb.`@event_name` AS event_name,
          etb.`@event_time` AS event_time,
          etb.`@event_name` AS event_name_group,
          etb.`@user_id` AS user_id
        FROM
          `test_event` AS etb
        WHERE
          etb.`@dt` between '2025-09-03 02:00:00' AND '2025-09-10 01:59:59'
          AND etb.`@event_name` = 'shop_buy'
          AND etb.`@user_id` IS NOT NULL
          AND etb.`@user_id` > '0'
      ) AS src
    GROUP BY
      src.event_date,
      src.user_id,
      src.event_name_group
  ) AS fwt
GROUP BY
  GROUPING SETS (
    (user_id),
    (user_id, event_date),
    (user_id, event_name_group),
    (user_id, event_date, event_name_group)
  );
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#43391 

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
seawinde added a commit to seawinde/doris that referenced this pull request Nov 11, 2025
@seawinde
[fix](nereids) Fix not in aggregate's output err after eliminate by u…
961cef6 
…niform when group sets exist (apache#56942)

Fix not in aggregate's output err after eliminate by uniform when group
sets exist

if query as following, would cause `ERROR 1105 (HY000): errCode = 2,
detailMessage = GROUPING_PREFIX_event_name_group not in aggregate's
output`

the pr fix this

```sql
SELECT
  CASE WHEN GROUPING(event_date) = 1 THEN '(TOTAL)' ELSE CAST(event_date AS VARCHAR) END AS event_date,
  user_id,
  MAX(conversion_level) AS conversion_level,
  CASE WHEN GROUPING(event_name_group) = 1 THEN '(TOTAL)' ELSE event_name_group END AS event_name_group
FROM
  (
    SELECT
      src.event_date,
      src.user_id,
      WINDOW_FUNNEL(
        3600 * 24 * 1,
        'default',
        src.event_time,
        src.event_name = 'shop_buy',
        src.event_name = 'shop_buy'
      ) AS conversion_level,
      src.event_name_group
    FROM
      (
        SELECT
          CAST(etb.`@dt` AS DATE) AS event_date,
          etb.`@event_name` AS event_name,
          etb.`@event_time` AS event_time,
          etb.`@event_name` AS event_name_group,
          etb.`@user_id` AS user_id
        FROM
          `test_event` AS etb
        WHERE
          etb.`@dt` between '2025-09-03 02:00:00' AND '2025-09-10 01:59:59'
          AND etb.`@event_name` = 'shop_buy'
          AND etb.`@user_id` IS NOT NULL
          AND etb.`@user_id` > '0'
      ) AS src
    GROUP BY
      src.event_date,
      src.user_id,
      src.event_name_group
  ) AS fwt
GROUP BY
  GROUPING SETS (
    (user_id),
    (user_id, event_date),
    (user_id, event_name_group),
    (user_id, event_date, event_name_group)
  );
```

Issue Number: close #xxx

Related PR: apache#43391

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@seawinde seawinde mentioned this pull request Nov 11, 2025
Merged
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 18, 2025
@seawinde @wyxxxcat
[fix](nereids) Fix not in aggregate's output err after eliminate by u…
3608ba7 
…niform when group sets exist (apache#56942)

Fix not in aggregate's output err after eliminate by uniform when group
sets exist

if query as following, would cause `ERROR 1105 (HY000): errCode = 2,
detailMessage = GROUPING_PREFIX_event_name_group not in aggregate's
output`

the pr fix this

```sql
SELECT
  CASE WHEN GROUPING(event_date) = 1 THEN '(TOTAL)' ELSE CAST(event_date AS VARCHAR) END AS event_date,
  user_id,
  MAX(conversion_level) AS conversion_level,
  CASE WHEN GROUPING(event_name_group) = 1 THEN '(TOTAL)' ELSE event_name_group END AS event_name_group
FROM
  (
    SELECT
      src.event_date,
      src.user_id,
      WINDOW_FUNNEL(
        3600 * 24 * 1,
        'default',
        src.event_time,
        src.event_name = 'shop_buy',
        src.event_name = 'shop_buy'
      ) AS conversion_level,
      src.event_name_group
    FROM
      (
        SELECT
          CAST(etb.`@dt` AS DATE) AS event_date,
          etb.`@event_name` AS event_name,
          etb.`@event_time` AS event_time,
          etb.`@event_name` AS event_name_group,
          etb.`@user_id` AS user_id
        FROM
          `test_event` AS etb
        WHERE
          etb.`@dt` between '2025-09-03 02:00:00' AND '2025-09-10 01:59:59'
          AND etb.`@event_name` = 'shop_buy'
          AND etb.`@user_id` IS NOT NULL
          AND etb.`@user_id` > '0'
      ) AS src
    GROUP BY
      src.event_date,
      src.user_id,
      src.event_name_group
  ) AS fwt
GROUP BY
  GROUPING SETS (
    (user_id),
    (user_id, event_date),
    (user_id, event_name_group),
    (user_id, event_date, event_name_group)
  );
```

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: apache#43391 

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow left review comments

@englefly englefly englefly approved these changes

@xzj7019 xzj7019 xzj7019 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@feiniaofeiafei @doris-robot @englefly @morrySnow @xzj7019

Comments