[fix](paimon)Support user-defined S3 config prefixes and unify to HDFS S3A protocol#57116

CalvinKirs · 2025-10-17T13:30:42Z

…

Previously, for FS-based Paimon catalogs, internal configuration translation was performed to ensure the storage layer used S3FileIO (which internally relied on Hadoop S3).

However, Paimon also allows users to specify S3-related options with various prefixes such as s3., s3a., or fs.s3a. in their configuration. The S3FileIO implementation in Paimon would automatically normalize these keys to the standard Hadoop prefix fs.s3a..

With the recent refactor, we have unified all object storage access to use the HDFS S3A protocol directly. Therefore, the system must now handle these legacy user-defined prefixes internally to ensure compatibility.

Before this change, users might define custom parameters like:

paimon.s3.list.version=1
paimon.s3.paging.maximum=100
paimon.fs.s3.read.ahead.buffer.size=1
paimon.s3a.replication.factor=3

After normalization, they are automatically converted to Hadoop-compatible S3A keys:

fs.s3a.list.version=1
fs.s3a.paging.maximum=100
fs.s3a.read.ahead.buffer.size=1
fs.s3a.replication.factor=3

Doc: apache/doris-website#3030

…FS S3A protocol Previously, for FS-based Paimon catalogs, internal configuration translation was performed to ensure the storage layer used S3FileIO (which internally relied on Hadoop S3). However, Paimon also allows users to specify S3-related options with various prefixes such as s3., s3a., or fs.s3a. in their configuration. The S3FileIO implementation in Paimon would automatically normalize these keys to the standard Hadoop prefix fs.s3a.. With the recent refactor, we have unified all object storage access to use the HDFS S3A protocol directly. Therefore, the system must now handle these legacy user-defined prefixes internally to ensure compatibility. ``` Before this change, users might define custom parameters like: paimon.s3.list.version=1 paimon.s3.paging.maximum=100 paimon.fs.s3.read.ahead.buffer.size=1 paimon.s3a.replication.factor=3 After normalization, they are automatically converted to Hadoop-compatible S3A keys: fs.s3a.list.version=1 fs.s3a.paging.maximum=100 fs.s3a.read.ahead.buffer.size=1 fs.s3a.replication.factor=3 ```

Thearas · 2025-10-17T13:30:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

CalvinKirs · 2025-10-20T01:36:06Z

run buildall

doris-robot · 2025-10-20T02:15:18Z

ClickBench: Total hot run time: 30.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a5b7912747fd22160dc4c5e0dedbb2acca8bb23, data reload: false

query1	0.06	0.04	0.04
query2	0.08	0.06	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.12
query5	0.28	0.26	0.27
query6	1.16	0.66	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.05
query9	0.63	0.53	0.52
query10	0.57	0.58	0.59
query11	0.16	0.12	0.11
query12	0.16	0.13	0.12
query13	0.63	0.62	0.63
query14	1.02	1.02	1.03
query15	0.90	0.86	0.87
query16	0.40	0.39	0.39
query17	1.03	1.07	1.05
query18	0.22	0.20	0.20
query19	1.96	1.84	1.86
query20	0.02	0.01	0.02
query21	15.47	0.91	0.57
query22	0.77	1.18	0.65
query23	14.98	1.41	0.68
query24	7.48	1.20	0.45
query25	0.48	0.22	0.06
query26	0.48	0.15	0.14
query27	0.09	0.06	0.06
query28	9.52	1.38	0.93
query29	12.60	3.96	3.38
query30	0.28	0.13	0.12
query31	2.83	0.59	0.39
query32	3.25	0.56	0.49
query33	3.01	3.07	3.11
query34	16.19	5.50	4.87
query35	4.94	4.93	4.92
query36	0.71	0.53	0.50
query37	0.10	0.08	0.07
query38	0.06	0.06	0.04
query39	0.04	0.03	0.03
query40	0.17	0.17	0.15
query41	0.08	0.03	0.04
query42	0.04	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 104.84 s
Total hot run time: 30.27 s

hello-stephen · 2025-10-20T02:50:16Z

FE UT Coverage Report

Increment line coverage 94.74% (18/19) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-10-20T07:02:18Z

FE Regression Coverage Report

Increment line coverage 57.89% (11/19) 🎉
Increment coverage report
Complete coverage report

CalvinKirs · 2025-10-20T11:51:10Z

run buildall

doris-robot · 2025-10-20T12:32:55Z

TPC-DS: Total hot run time: 190384 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6315f882a1814cf65ddf9dd3322f5af9792b5400, data reload: false

query1	1052	438	401	401
query2	6558	1723	1705	1705
query3	6745	229	222	222
query4	26359	23722	23612	23612
query5	5022	623	499	499
query6	353	260	241	241
query7	4672	495	299	299
query8	313	267	260	260
query9	8734	2573	2547	2547
query10	537	355	307	307
query11	15727	15117	14839	14839
query12	195	119	119	119
query13	1682	567	445	445
query14	12624	9276	9268	9268
query15	254	198	180	180
query16	7788	702	513	513
query17	1606	824	621	621
query18	2129	456	354	354
query19	285	221	183	183
query20	137	131	134	131
query21	229	144	120	120
query22	4745	4718	4647	4647
query23	35006	34282	33855	33855
query24	8277	2491	2501	2491
query25	634	531	485	485
query26	1578	274	174	174
query27	2781	527	386	386
query28	4450	2258	2221	2221
query29	818	629	508	508
query30	314	235	202	202
query31	976	870	761	761
query32	90	71	71	71
query33	629	363	327	327
query34	789	853	521	521
query35	848	832	760	760
query36	930	972	885	885
query37	122	109	87	87
query38	3498	3555	3541	3541
query39	1474	1423	1411	1411
query40	214	125	113	113
query41	60	59	58	58
query42	123	113	108	108
query43	488	495	468	468
query44	1315	818	823	818
query45	183	178	172	172
query46	848	1020	646	646
query47	1763	1829	1693	1693
query48	411	425	308	308
query49	763	521	403	403
query50	638	689	420	420
query51	3940	3976	3937	3937
query52	113	112	99	99
query53	233	272	201	201
query54	601	597	562	562
query55	96	86	84	84
query56	325	316	306	306
query57	1148	1192	1099	1099
query58	284	284	277	277
query59	2539	2613	2524	2524
query60	361	348	324	324
query61	156	158	155	155
query62	795	744	679	679
query63	232	192	197	192
query64	4400	1210	899	899
query65	4060	3956	4016	3956
query66	1076	423	355	355
query67	15317	15197	15136	15136
query68	9567	899	599	599
query69	502	406	296	296
query70	1314	1273	1285	1273
query71	518	322	326	322
query72	5537	4928	4820	4820
query73	693	570	364	364
query74	9336	9051	8640	8640
query75	4501	3349	2851	2851
query76	4254	1153	751	751
query77	1117	399	330	330
query78	9625	9805	8879	8879
query79	2701	832	586	586
query80	701	571	512	512
query81	498	261	234	234
query82	496	172	132	132
query83	293	270	325	270
query84	303	106	97	97
query85	877	474	425	425
query86	351	310	326	310
query87	3769	3715	3569	3569
query88	3050	2258	2227	2227
query89	407	325	294	294
query90	2049	220	220	220
query91	167	175	133	133
query92	81	77	66	66
query93	1266	981	644	644
query94	696	430	347	347
query95	408	333	320	320
query96	483	584	280	280
query97	2929	2987	2880	2880
query98	240	209	208	208
query99	1415	1413	1314	1314
Total cold run time: 283586 ms
Total hot run time: 190384 ms

doris-robot · 2025-10-20T12:38:26Z

ClickBench: Total hot run time: 30.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6315f882a1814cf65ddf9dd3322f5af9792b5400, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.06
query3	0.26	0.09	0.09
query4	1.62	0.12	0.12
query5	0.27	0.27	0.25
query6	1.17	0.67	0.64
query7	0.03	0.03	0.02
query8	0.05	0.05	0.05
query9	0.63	0.52	0.53
query10	0.57	0.58	0.57
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.67	0.62	0.61
query14	1.03	1.03	1.04
query15	0.86	0.87	0.87
query16	0.40	0.40	0.41
query17	1.05	1.06	1.06
query18	0.23	0.20	0.20
query19	1.93	1.84	1.80
query20	0.02	0.01	0.02
query21	15.45	0.92	0.58
query22	0.75	1.16	0.75
query23	14.84	1.37	0.62
query24	6.86	1.34	0.63
query25	0.46	0.18	0.20
query26	0.67	0.16	0.13
query27	0.07	0.05	0.06
query28	10.05	1.36	0.94
query29	12.58	3.94	3.24
query30	0.28	0.14	0.11
query31	2.86	0.61	0.39
query32	3.24	0.56	0.48
query33	3.14	3.17	3.10
query34	16.14	5.43	4.90
query35	4.99	4.93	4.92
query36	0.71	0.52	0.53
query37	0.10	0.07	0.08
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.9 s
Total hot run time: 30.48 s

CalvinKirs · 2025-10-20T14:13:37Z

run buildall

doris-robot · 2025-10-20T14:50:25Z

ClickBench: Total hot run time: 31.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0397c65204f936b71a58ab36ccf8dc5aa207a0e5, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.06	0.06
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.28	0.26	0.25
query6	1.20	0.66	0.65
query7	0.03	0.03	0.03
query8	0.06	0.05	0.04
query9	0.63	0.53	0.54
query10	0.59	0.57	0.58
query11	0.17	0.12	0.11
query12	0.15	0.12	0.12
query13	0.65	0.62	0.62
query14	1.04	1.07	1.06
query15	0.90	0.89	0.87
query16	0.41	0.41	0.40
query17	1.11	1.11	1.09
query18	0.23	0.20	0.20
query19	1.97	1.84	1.94
query20	0.02	0.01	0.01
query21	15.42	0.95	0.57
query22	0.76	1.30	1.02
query23	14.70	1.43	0.63
query24	7.10	1.60	1.05
query25	0.49	0.19	0.15
query26	0.61	0.16	0.14
query27	0.06	0.06	0.06
query28	9.58	1.39	0.96
query29	12.57	4.03	3.28
query30	0.29	0.13	0.13
query31	2.81	0.62	0.38
query32	3.23	0.55	0.48
query33	3.05	3.14	3.16
query34	16.19	5.55	4.86
query35	5.00	4.92	4.92
query36	0.73	0.53	0.53
query37	0.10	0.08	0.07
query38	0.06	0.05	0.05
query39	0.03	0.04	0.03
query40	0.17	0.15	0.14
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 104.56 s
Total hot run time: 31.39 s

hello-stephen · 2025-10-20T15:31:20Z

FE UT Coverage Report

Increment line coverage 87.27% (48/55) 🎉
Increment coverage report
Complete coverage report

CalvinKirs · 2025-10-21T03:12:24Z

run buildall

doris-robot · 2025-10-21T03:47:09Z

ClickBench: Total hot run time: 31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 81fb14597af5cd95d363c78fa07353dc8055c580, data reload: false

query1	0.05	0.05	0.05
query2	0.08	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.30	0.27	0.25
query6	1.21	0.68	0.65
query7	0.03	0.03	0.03
query8	0.07	0.05	0.05
query9	0.63	0.53	0.51
query10	0.58	0.58	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.65	0.62	0.63
query14	1.05	1.04	1.03
query15	0.87	0.86	0.88
query16	0.42	0.40	0.42
query17	1.07	1.05	1.08
query18	0.23	0.21	0.21
query19	1.97	1.87	1.85
query20	0.02	0.01	0.02
query21	15.43	0.94	0.58
query22	0.75	1.19	0.89
query23	14.74	1.37	0.61
query24	7.06	0.99	0.90
query25	0.48	0.20	0.13
query26	0.65	0.16	0.14
query27	0.07	0.05	0.06
query28	10.09	1.36	0.93
query29	12.63	4.08	3.43
query30	0.28	0.16	0.11
query31	2.83	0.60	0.40
query32	3.27	0.56	0.50
query33	3.04	3.12	3.09
query34	16.16	5.54	4.83
query35	4.92	4.95	4.92
query36	0.69	0.52	0.51
query37	0.11	0.07	0.07
query38	0.07	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.08	0.03	0.04
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 105.05 s
Total hot run time: 31 s

hello-stephen · 2025-10-21T04:22:57Z

FE UT Coverage Report

Increment line coverage 87.04% (47/54) 🎉
Increment coverage report
Complete coverage report

...e/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java

…metastore/AbstractPaimonProperties.java Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>

CalvinKirs · 2025-10-21T06:42:56Z

run buildall

doris-robot · 2025-10-21T07:26:17Z

TPC-DS: Total hot run time: 190318 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 239be306af1ab89bf8821cfd05a7c477badef856, data reload: false

query1	1060	430	398	398
query2	6552	1690	1692	1690
query3	6781	229	237	229
query4	26361	23603	23952	23603
query5	4435	634	488	488
query6	328	254	249	249
query7	4654	502	302	302
query8	310	267	276	267
query9	8722	2613	2622	2613
query10	503	336	289	289
query11	15849	15182	14898	14898
query12	192	121	117	117
query13	1689	559	452	452
query14	13068	9296	9435	9296
query15	262	190	175	175
query16	7840	683	485	485
query17	1640	758	765	758
query18	2410	487	344	344
query19	303	245	178	178
query20	143	149	137	137
query21	217	132	116	116
query22	4664	4714	4470	4470
query23	34428	33772	33740	33740
query24	7123	2648	2436	2436
query25	564	530	469	469
query26	1286	298	166	166
query27	2550	527	363	363
query28	4519	2269	2248	2248
query29	788	660	524	524
query30	405	232	202	202
query31	926	854	753	753
query32	139	71	78	71
query33	600	405	345	345
query34	844	848	545	545
query35	850	871	789	789
query36	1014	1093	939	939
query37	123	108	79	79
query38	3505	3600	3504	3504
query39	1525	1400	1408	1400
query40	213	126	114	114
query41	61	58	56	56
query42	121	114	116	114
query43	477	499	466	466
query44	1217	742	732	732
query45	187	182	175	175
query46	878	997	632	632
query47	1755	1831	1706	1706
query48	406	422	321	321
query49	767	501	414	414
query50	657	690	414	414
query51	3840	3947	3879	3879
query52	107	109	101	101
query53	229	269	204	204
query54	606	588	530	530
query55	89	84	90	84
query56	321	325	300	300
query57	1177	1182	1109	1109
query58	284	268	272	268
query59	2471	2566	2620	2566
query60	362	340	330	330
query61	156	153	152	152
query62	802	777	660	660
query63	234	190	189	189
query64	4502	1247	893	893
query65	4061	3945	3973	3945
query66	1065	444	336	336
query67	15471	15151	15093	15093
query68	9075	885	608	608
query69	484	343	296	296
query70	1380	1276	1289	1276
query71	511	344	321	321
query72	6019	4799	4775	4775
query73	704	591	358	358
query74	9208	8890	8877	8877
query75	4469	3411	2820	2820
query76	3748	1172	738	738
query77	817	412	316	316
query78	9615	9639	8855	8855
query79	3842	847	579	579
query80	784	581	504	504
query81	506	270	227	227
query82	690	170	139	139
query83	303	269	252	252
query84	297	120	91	91
query85	903	549	418	418
query86	340	300	280	280
query87	3766	3798	3709	3709
query88	3321	2270	2245	2245
query89	417	330	315	315
query90	2065	219	213	213
query91	166	165	134	134
query92	87	63	65	63
query93	2195	996	639	639
query94	701	449	349	349
query95	416	331	314	314
query96	486	578	284	284
query97	2956	3005	2879	2879
query98	245	209	207	207
query99	1457	1409	1327	1327
Total cold run time: 283661 ms
Total hot run time: 190318 ms

doris-robot · 2025-10-21T07:31:28Z

ClickBench: Total hot run time: 28.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 239be306af1ab89bf8821cfd05a7c477badef856, data reload: false

query1	0.05	0.05	0.04
query2	0.09	0.06	0.05
query3	0.25	0.09	0.09
query4	1.61	0.12	0.12
query5	0.28	0.26	0.25
query6	1.18	0.66	0.64
query7	0.03	0.03	0.03
query8	0.06	0.05	0.05
query9	0.64	0.52	0.52
query10	0.58	0.58	0.57
query11	0.17	0.12	0.12
query12	0.17	0.12	0.12
query13	0.64	0.62	0.62
query14	1.03	1.03	1.03
query15	0.87	0.84	0.88
query16	0.41	0.40	0.40
query17	1.09	1.03	1.07
query18	0.21	0.20	0.20
query19	1.97	1.88	1.83
query20	0.01	0.02	0.01
query21	15.45	0.18	0.13
query22	5.17	0.06	0.04
query23	15.66	0.26	0.10
query24	3.44	0.82	0.32
query25	0.07	0.06	0.06
query26	0.13	0.13	0.13
query27	0.06	0.06	0.06
query28	3.63	1.15	0.92
query29	12.55	4.01	3.27
query30	0.29	0.13	0.11
query31	2.82	0.57	0.38
query32	3.24	0.58	0.49
query33	3.13	3.10	3.10
query34	16.13	5.52	4.87
query35	4.91	4.94	4.93
query36	0.70	0.51	0.50
query37	0.09	0.06	0.07
query38	0.07	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.15	0.14
query41	0.08	0.03	0.04
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 99.25 s
Total hot run time: 28.31 s

hello-stephen · 2025-10-21T09:27:23Z

FE Regression Coverage Report

Increment line coverage 70.37% (38/54) 🎉
Increment coverage report
Complete coverage report

…er-paimon-list.version # Conflicts: # fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/AzureProperties.java

CalvinKirs · 2025-10-23T02:27:45Z

run buildall

doris-robot · 2025-10-23T03:18:20Z

TPC-DS: Total hot run time: 190875 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f4209b3b82dab19d68f6cb8c073b6fbc10fe44f8, data reload: false

query1	1067	415	402	402
query2	6567	1719	1738	1719
query3	6775	233	225	225
query4	26553	23537	23476	23476
query5	4807	643	462	462
query6	340	269	255	255
query7	4650	523	306	306
query8	307	287	250	250
query9	8695	2582	2583	2582
query10	502	352	279	279
query11	15784	15838	15697	15697
query12	201	121	122	121
query13	2416	575	458	458
query14	12364	9727	9559	9559
query15	213	226	176	176
query16	7930	715	536	536
query17	1965	823	670	670
query18	2175	443	356	356
query19	243	229	193	193
query20	143	129	149	129
query21	278	156	139	139
query22	4818	4674	4567	4567
query23	35194	33560	33294	33294
query24	8334	2428	2425	2425
query25	558	510	452	452
query26	1231	280	163	163
query27	2701	491	355	355
query28	4312	2162	2140	2140
query29	788	614	476	476
query30	312	225	190	190
query31	969	842	736	736
query32	77	70	71	70
query33	589	367	332	332
query34	798	872	548	548
query35	794	821	753	753
query36	967	1015	881	881
query37	121	116	80	80
query38	3530	3588	3533	3533
query39	1452	1429	1427	1427
query40	232	126	120	120
query41	65	58	57	57
query42	127	112	107	107
query43	495	503	469	469
query44	1217	743	747	743
query45	190	182	179	179
query46	880	1008	658	658
query47	1757	1767	1705	1705
query48	400	432	329	329
query49	777	513	438	438
query50	648	714	410	410
query51	3861	3946	4259	3946
query52	116	107	100	100
query53	232	272	196	196
query54	597	591	541	541
query55	91	89	88	88
query56	343	348	292	292
query57	1177	1164	1125	1125
query58	290	283	277	277
query59	2629	2651	2637	2637
query60	352	348	352	348
query61	159	156	149	149
query62	820	717	700	700
query63	243	197	199	197
query64	4386	1175	859	859
query65	4060	3970	3937	3937
query66	1121	435	334	334
query67	15357	15227	15167	15167
query68	9608	910	620	620
query69	487	327	287	287
query70	1366	1247	1230	1230
query71	513	348	311	311
query72	5495	4801	4802	4801
query73	691	568	355	355
query74	8850	8870	8947	8870
query75	4425	3355	2866	2866
query76	3710	1171	743	743
query77	915	411	326	326
query78	9500	9718	8901	8901
query79	3225	841	587	587
query80	699	562	484	484
query81	493	264	225	225
query82	270	165	140	140
query83	297	269	295	269
query84	298	115	93	93
query85	873	467	423	423
query86	338	298	289	289
query87	3743	3745	3612	3612
query88	2861	2272	2230	2230
query89	432	339	300	300
query90	2169	225	224	224
query91	162	173	135	135
query92	87	67	66	66
query93	2321	972	632	632
query94	691	465	329	329
query95	415	315	308	308
query96	490	581	292	292
query97	2921	2963	2887	2887
query98	243	214	212	212
query99	1440	1437	1308	1308
Total cold run time: 284170 ms
Total hot run time: 190875 ms

doris-robot · 2025-10-23T03:23:29Z

ClickBench: Total hot run time: 27.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f4209b3b82dab19d68f6cb8c073b6fbc10fe44f8, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.26	0.08	0.08
query4	1.60	0.12	0.11
query5	0.27	0.26	0.27
query6	1.18	0.67	0.65
query7	0.04	0.03	0.02
query8	0.05	0.04	0.04
query9	0.62	0.52	0.52
query10	0.59	0.58	0.58
query11	0.17	0.12	0.11
query12	0.17	0.13	0.12
query13	0.62	0.60	0.61
query14	1.00	1.00	1.03
query15	0.87	0.84	0.87
query16	0.40	0.40	0.42
query17	1.04	1.07	1.02
query18	0.22	0.20	0.20
query19	1.85	1.86	1.80
query20	0.01	0.01	0.02
query21	15.43	0.18	0.13
query22	4.97	0.07	0.05
query23	15.62	0.27	0.10
query24	2.96	0.61	0.98
query25	0.07	0.06	0.06
query26	0.14	0.13	0.13
query27	0.07	0.06	0.06
query28	4.95	1.15	0.93
query29	12.60	3.88	3.26
query30	0.29	0.14	0.12
query31	2.82	0.59	0.40
query32	3.24	0.55	0.47
query33	3.06	3.11	3.04
query34	15.84	5.21	4.57
query35	4.58	4.61	4.56
query36	0.68	0.51	0.50
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.04	0.03
query40	0.17	0.15	0.15
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 99.01 s
Total hot run time: 27.83 s

github-actions · 2025-10-23T05:17:23Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-10-23T05:17:25Z

PR approved by anyone and no changes requested.

hello-stephen · 2025-10-23T05:34:09Z

FE Regression Coverage Report

Increment line coverage 73.08% (38/52) 🎉
Increment coverage report
Complete coverage report

…S S3A protocol (apache#57116) … Previously, for FS-based Paimon catalogs, internal configuration translation was performed to ensure the storage layer used S3FileIO (which internally relied on Hadoop S3). However, Paimon also allows users to specify S3-related options with various prefixes such as s3., s3a., or fs.s3a. in their configuration. The S3FileIO implementation in Paimon would automatically normalize these keys to the standard Hadoop prefix fs.s3a.. With the recent refactor, we have unified all object storage access to use the HDFS S3A protocol directly. Therefore, the system must now handle these legacy user-defined prefixes internally to ensure compatibility. ``` Before this change, users might define custom parameters like: paimon.s3.list.version=1 paimon.s3.paging.maximum=100 paimon.fs.s3.read.ahead.buffer.size=1 paimon.s3a.replication.factor=3 After normalization, they are automatically converted to Hadoop-compatible S3A keys: fs.s3a.list.version=1 fs.s3a.paging.maximum=100 fs.s3a.read.ahead.buffer.size=1 fs.s3a.replication.factor=3 ``` --------- Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>

…S S3A protocol (apache#57116) … Previously, for FS-based Paimon catalogs, internal configuration translation was performed to ensure the storage layer used S3FileIO (which internally relied on Hadoop S3). However, Paimon also allows users to specify S3-related options with various prefixes such as s3., s3a., or fs.s3a. in their configuration. The S3FileIO implementation in Paimon would automatically normalize these keys to the standard Hadoop prefix fs.s3a.. With the recent refactor, we have unified all object storage access to use the HDFS S3A protocol directly. Therefore, the system must now handle these legacy user-defined prefixes internally to ensure compatibility. ``` Before this change, users might define custom parameters like: paimon.s3.list.version=1 paimon.s3.paging.maximum=100 paimon.fs.s3.read.ahead.buffer.size=1 paimon.s3a.replication.factor=3 After normalization, they are automatically converted to Hadoop-compatible S3A keys: fs.s3a.list.version=1 fs.s3a.paging.maximum=100 fs.s3a.read.ahead.buffer.size=1 fs.s3a.replication.factor=3 ``` --------- Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com> (cherry picked from commit d739136)

…unify to HDFS S3A protocol (#57116) (#57526) cp #57116

…S S3A protocol apache#57116

…unify to HDFS S3A protocol #57116 (#58544) #57116 #58467 #57933

CalvinKirs added dev/3.1.x dev/4.0.x labels Oct 17, 2025

CalvinKirs changed the title ~~[fix][PAIMON] Support user-defined S3 config prefixes and unify to HDFS S3A protocol~~ [fix](paimon)Support user-defined S3 config prefixes and unify to HDFS S3A protocol Oct 20, 2025

CalvinKirs added 2 commits October 20, 2025 16:53

Merge branch 'master' into master-paimon-list.version

8c92858

fix

6315f88

fix

0397c65

fix

81fb145

morningman reviewed Oct 21, 2025

View reviewed changes

...e/src/main/java/org/apache/doris/datasource/property/metastore/AbstractPaimonProperties.java Outdated Show resolved Hide resolved

Update fe/fe-core/src/main/java/org/apache/doris/datasource/property/…

239be30

…metastore/AbstractPaimonProperties.java Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>

CalvinKirs marked this pull request as draft October 21, 2025 11:21

Merge remote-tracking branch 'refs/remotes/upstream/master' into mast…

a6508dd

…er-paimon-list.version # Conflicts: # fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/AzureProperties.java

checkstyle

f4209b3

morningman approved these changes Oct 23, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 23, 2025

github-actions bot added the reviewed label Oct 23, 2025

hubgeter approved these changes Oct 23, 2025

View reviewed changes

CalvinKirs merged commit d739136 into apache:master Oct 23, 2025
27 of 28 checks passed

github-actions bot added dev/3.1.x-conflict dev/4.0.x-conflict labels Oct 23, 2025

shuke987 added the p0_b label Oct 28, 2025

CalvinKirs deleted the master-paimon-list.version branch October 30, 2025 06:31

CalvinKirs removed the dev/3.1.x-conflict label Oct 30, 2025

CalvinKirs mentioned this pull request Oct 30, 2025

branch-3.1: [fix](paimon)Support user-defined S3 config prefixes and unify to HDFS S3A protocol (#57116) #57526

Merged

morningman pushed a commit that referenced this pull request Oct 31, 2025

branch-3.1: [fix](paimon)Support user-defined S3 config prefixes and …

0a687a6

…unify to HDFS S3A protocol (#57116) (#57526) cp #57116

morningman added dev/3.1.3-merged and removed dev/3.1.x labels Oct 31, 2025

CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Dec 1, 2025

[fix](paimon)Support user-defined S3 config prefixes and unify to HDF…

e9d4887

…S S3A protocol apache#57116

CalvinKirs mentioned this pull request Dec 1, 2025

branch-4.0: [fix](paimon)Support user-defined S3 config prefixes and unify to HDFS S3A protocol #57116 #58544

Merged

yiguolei pushed a commit that referenced this pull request Dec 2, 2025

branch-4.0: [fix](paimon)Support user-defined S3 config prefixes and …

7ce9386

…unify to HDFS S3A protocol #57116 (#58544) #57116 #58467 #57933

yiguolei added dev/4.0.2-merged and removed dev/4.0.x dev/4.0.x-conflict labels Dec 2, 2025

yiguolei mentioned this pull request Dec 2, 2025

4.0.2 Release Notes #58605

Open

Conversation

CalvinKirs commented Oct 17, 2025 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Thearas commented Oct 17, 2025

Uh oh!

CalvinKirs commented Oct 20, 2025

Uh oh!

doris-robot commented Oct 20, 2025

Uh oh!

hello-stephen commented Oct 20, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Oct 20, 2025

FE Regression Coverage Report

Uh oh!

CalvinKirs commented Oct 20, 2025

Uh oh!

doris-robot commented Oct 20, 2025

Uh oh!

doris-robot commented Oct 20, 2025

Uh oh!

CalvinKirs commented Oct 20, 2025

Uh oh!

doris-robot commented Oct 20, 2025

Uh oh!

hello-stephen commented Oct 20, 2025

FE UT Coverage Report

Uh oh!

CalvinKirs commented Oct 21, 2025

Uh oh!

doris-robot commented Oct 21, 2025

Uh oh!

hello-stephen commented Oct 21, 2025

FE UT Coverage Report

Uh oh!

Uh oh!

CalvinKirs commented Oct 21, 2025

Uh oh!

doris-robot commented Oct 21, 2025

Uh oh!

doris-robot commented Oct 21, 2025

Uh oh!

hello-stephen commented Oct 21, 2025

FE Regression Coverage Report

Uh oh!

CalvinKirs commented Oct 23, 2025

Uh oh!

doris-robot commented Oct 23, 2025

Uh oh!

doris-robot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

hello-stephen commented Oct 23, 2025

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

CalvinKirs commented Oct 17, 2025 •

edited by morningman

Loading