Skip to content

[fix]handle loading iceberg partitions by auth#57988

Merged
CalvinKirs merged 3 commits intoapache:masterfrom
zhaorongsheng:bugfix_iceberg_krb5_error
Nov 14, 2025
Merged

[fix]handle loading iceberg partitions by auth#57988
CalvinKirs merged 3 commits intoapache:masterfrom
zhaorongsheng:bugfix_iceberg_krb5_error

Conversation

@zhaorongsheng
Copy link
Contributor

@zhaorongsheng zhaorongsheng commented Nov 13, 2025

What problem does this PR solve?

Issue Number: close #57987

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@zhaorongsheng
handle loading iceberg partitions by auth
50cca28 
Wrap loading of iceberg partitions in try-catch block to handle exceptions and log errors.
@zhaorongsheng
Refactor storage properties retrieval in IcebergScanNode
5661187
@Thearas
Copy link
Contributor

Thearas commented Nov 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei yiguolei added dev/4.0.x usercase Important user case type label dev/3.1.x labels Nov 13, 2025
@yiguolei
Copy link
Contributor

yiguolei commented Nov 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 13, 2025

TPC-H: Total hot run time: 34402 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8a9b3a80202a3ff92289f4dc97218a6a1cd150cb, data reload: false

------ Round 1 ----------------------------------
q1	17598	5223	5167	5167
q2	2040	332	215	215
q3	10258	1287	741	741
q4	10260	920	376	376
q5	7536	2432	2353	2353
q6	184	167	137	137
q7	904	777	635	635
q8	9367	1307	1069	1069
q9	6877	5213	5112	5112
q10	6896	2201	1790	1790
q11	489	312	290	290
q12	345	374	245	245
q13	17774	3679	3020	3020
q14	233	247	215	215
q15	589	519	514	514
q16	1040	1008	945	945
q17	598	868	363	363
q18	7495	7135	7098	7098
q19	1455	945	561	561
q20	349	345	239	239
q21	4108	3249	2368	2368
q22	1090	1047	949	949
Total cold run time: 107485 ms
Total hot run time: 34402 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5177	5125	5136	5125
q2	250	321	238	238
q3	2220	2663	2302	2302
q4	1354	1760	1333	1333
q5	4202	4490	4549	4490
q6	201	165	127	127
q7	2053	1979	1847	1847
q8	2699	2578	2634	2578
q9	7385	7292	7324	7292
q10	3037	3306	2848	2848
q11	627	541	528	528
q12	686	781	598	598
q13	3480	3981	3449	3449
q14	285	301	268	268
q15	560	506	506	506
q16	1081	1099	1028	1028
q17	1240	1659	1413	1413
q18	7866	7659	7615	7615
q19	850	825	907	825
q20	2027	2030	1903	1903
q21	4769	4357	4302	4302
q22	1072	1048	1003	1003
Total cold run time: 53121 ms
Total hot run time: 51618 ms

@doris-robot
Copy link

doris-robot commented Nov 13, 2025

TPC-DS: Total hot run time: 187776 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8a9b3a80202a3ff92289f4dc97218a6a1cd150cb, data reload: false

query1	1040	400	387	387
query2	6553	1681	1681	1681
query3	6771	228	223	223
query4	26239	23858	23007	23007
query5	4430	626	475	475
query6	343	247	234	234
query7	4656	517	292	292
query8	322	263	253	253
query9	8746	2594	2608	2594
query10	512	344	309	309
query11	15757	15074	14863	14863
query12	185	122	115	115
query13	1695	561	467	467
query14	10963	9198	9229	9198
query15	204	192	173	173
query16	7385	691	540	540
query17	1250	785	641	641
query18	2015	423	342	342
query19	218	201	184	184
query20	131	126	127	126
query21	220	137	118	118
query22	3968	4127	4101	4101
query23	33938	33316	33034	33034
query24	8444	2422	2429	2422
query25	642	571	477	477
query26	1243	272	167	167
query27	2750	494	355	355
query28	4394	2223	2206	2206
query29	873	643	516	516
query30	307	239	198	198
query31	904	804	719	719
query32	94	76	78	76
query33	619	387	371	371
query34	818	852	559	559
query35	820	829	785	785
query36	947	981	905	905
query37	126	110	87	87
query38	3507	3452	3459	3452
query39	1507	1418	1429	1418
query40	220	128	119	119
query41	64	62	66	62
query42	128	113	111	111
query43	506	497	474	474
query44	1234	730	735	730
query45	189	179	179	179
query46	875	983	640	640
query47	1790	1798	1693	1693
query48	392	420	311	311
query49	781	516	426	426
query50	640	671	402	402
query51	3953	3982	3828	3828
query52	108	110	103	103
query53	233	269	204	204
query54	309	317	301	301
query55	85	88	81	81
query56	324	302	314	302
query57	1202	1189	1109	1109
query58	297	279	278	278
query59	2583	2655	2568	2568
query60	342	368	337	337
query61	157	163	157	157
query62	790	737	674	674
query63	234	193	196	193
query64	4531	1161	846	846
query65	4046	3957	3960	3957
query66	1170	440	337	337
query67	15294	15248	14970	14970
query68	8353	901	597	597
query69	478	337	282	282
query70	1310	1290	1258	1258
query71	492	363	304	304
query72	6054	4886	4893	4886
query73	658	578	361	361
query74	8817	9086	8761	8761
query75	3972	3303	2829	2829
query76	3815	1153	728	728
query77	816	391	309	309
query78	9546	9764	8941	8941
query79	2150	845	590	590
query80	645	560	522	522
query81	511	257	232	232
query82	462	167	139	139
query83	281	269	256	256
query84	248	120	92	92
query85	886	476	438	438
query86	392	311	300	300
query87	3698	3737	3592	3592
query88	3625	2258	2223	2223
query89	396	324	291	291
query90	1957	224	220	220
query91	175	166	142	142
query92	83	69	62	62
query93	1808	979	642	642
query94	712	454	334	334
query95	415	327	315	315
query96	482	575	288	288
query97	2922	2988	2874	2874
query98	252	214	211	211
query99	1369	1389	1295	1295
Total cold run time: 275793 ms
Total hot run time: 187776 ms

@doris-robot
Copy link

doris-robot commented Nov 13, 2025

ClickBench: Total hot run time: 27.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8a9b3a80202a3ff92289f4dc97218a6a1cd150cb, data reload: false

query1	0.06	0.05	0.04
query2	0.10	0.05	0.05
query3	0.25	0.08	0.09
query4	1.61	0.11	0.10
query5	0.27	0.25	0.25
query6	1.16	0.66	0.64
query7	0.03	0.03	0.03
query8	0.05	0.05	0.04
query9	0.57	0.51	0.51
query10	0.58	0.56	0.58
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.62	0.60	0.60
query14	1.00	0.99	1.00
query15	0.84	0.84	0.85
query16	0.39	0.40	0.39
query17	1.01	1.03	1.02
query18	0.21	0.21	0.20
query19	1.97	1.82	1.79
query20	0.02	0.02	0.02
query21	15.44	0.18	0.13
query22	5.19	0.08	0.05
query23	15.65	0.28	0.10
query24	2.56	0.84	0.88
query25	0.11	0.07	0.06
query26	0.15	0.13	0.14
query27	0.07	0.05	0.05
query28	5.05	1.15	0.94
query29	12.62	3.92	3.23
query30	0.28	0.15	0.12
query31	2.82	0.58	0.39
query32	3.22	0.54	0.48
query33	3.02	3.06	3.08
query34	15.92	5.21	4.55
query35	4.58	4.65	4.59
query36	0.69	0.51	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 98.98 s
Total hot run time: 27.96 s

CalvinKirs
CalvinKirs reviewed Nov 13, 2025
View reviewed changes
Copy link
Member

@CalvinKirs CalvinKirs left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thx, could you add a test? FYI iceberg_on_hms_and_filesystem_and_dlf.groovy

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 13, 2025

FE Regression Coverage Report

Increment line coverage 50.00% (5/10) 🎉
Increment coverage report
Complete coverage report

@zhaorongsheng
Copy link
Contributor Author

zhaorongsheng commented Nov 14, 2025

Thx, could you add a test? FYI iceberg_on_hms_and_filesystem_and_dlf.groovy

ok

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 14, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 14, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 14, 2025

PR approved by anyone and no changes requested.

@CalvinKirs CalvinKirs merged commit d1f036d into apache:master Nov 14, 2025
34 of 35 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 14, 2025
@zhaorongsheng
[fix]handle loading iceberg partitions by auth (#57988)
2361cbd 
### What problem does this PR solve?

Issue Number: close #57987



### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot mentioned this pull request Nov 14, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Nov 14, 2025
@zhaorongsheng
[fix]handle loading iceberg partitions by auth (#57988)
4ce9e82 
### What problem does this PR solve?

Issue Number: close #57987



### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot mentioned this pull request Nov 14, 2025
Merged
yiguolei pushed a commit that referenced this pull request Nov 14, 2025
@github-actions @zhaorongsheng
branch-4.0: [fix]handle loading iceberg partitions by auth #57988 (#5…
7355f76 
…8021)

Cherry-picked from #57988

Co-authored-by: zhaorongsheng <zhaorongsheng@users.noreply.github.com>
morrySnow pushed a commit that referenced this pull request Nov 25, 2025
@github-actions @zhaorongsheng
branch-3.1: [fix]handle loading iceberg partitions by auth #57988 (#5…
7190d05 
…8020)

Cherry-picked from #57988

Co-authored-by: zhaorongsheng <zhaorongsheng@users.noreply.github.com>
@yiguolei yiguolei mentioned this pull request Dec 2, 2025
Open
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@zhaorongsheng @nagisa-kunhah
[fix]handle loading iceberg partitions by auth (apache#57988)
5cb1810 
### What problem does this PR solve?

Issue Number: close apache#57987



### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] Failed to read iceberg table by hms

8 participants

@zhaorongsheng @Thearas @yiguolei @doris-robot @hello-stephen @morningman @CalvinKirs @morrySnow

Comments