Skip to content

[[fix](rest-s3)Set AWS Request Checksum Calculation only if not set void issues on non-S3 storage#58467

Merged
morningman merged 1 commit intoapache:masterfrom
CalvinKirs:master-rest-s3
Nov 28, 2025
Merged

[[fix](rest-s3)Set AWS Request Checksum Calculation only if not set void issues on non-S3 storage#58467
morningman merged 1 commit intoapache:masterfrom
CalvinKirs:master-rest-s3

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Nov 27, 2025 •
edited
Loading

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.

According to the official discussion AWS SDK v2 Discussion aws/aws-sdk-java-v2#5802

By default, the SDK may automatically calculate checksums when required.

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.

For non-S3 object storage (such as COS, OSS, OBS), using this default value may cause errors, for example:

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Nov 27, 2025

run buildall

@CalvinKirs
[fix](rest-s3)Set AWS Request Checksum Calculation only if not set, a…
5251cb7 
…void issues on non-S3 storage

AWS SDK for Java v2 uses the system property AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request payload checksum calculation.

According to the official discussion AWS SDK v2 Discussion apache#5802
:

By default, the SDK may automatically calculate checksums when required.

For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums are calculated only when necessary, providing both compatibility and performance benefits.

aws-chunked encoding is not supported with the specified x-amz-content-sha256 value

because these services do not support AWS-specific chunked encoding or SHA256 checksum.
@CalvinKirs
Copy link
Member Author

CalvinKirs commented Nov 27, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-H: Total hot run time: 35039 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5251cb76341f0dcbb563308df84579a69c75db90, data reload: false

------ Round 1 ----------------------------------
q1	17684	5027	4872	4872
q2	2063	320	201	201
q3	10237	1313	752	752
q4	10231	913	378	378
q5	7535	2522	2226	2226
q6	191	183	139	139
q7	961	782	632	632
q8	9370	1417	1043	1043
q9	7076	5366	5308	5308
q10	6894	2245	1797	1797
q11	515	316	302	302
q12	368	378	217	217
q13	17775	3671	3006	3006
q14	235	243	216	216
q15	602	511	517	511
q16	1053	1001	967	967
q17	617	826	456	456
q18	7976	7480	7631	7480
q19	1221	1027	590	590
q20	362	349	232	232
q21	4270	3611	2674	2674
q22	1114	1069	1040	1040
Total cold run time: 108350 ms
Total hot run time: 35039 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5290	5268	5116	5116
q2	333	419	327	327
q3	2340	2926	2430	2430
q4	1390	1841	1433	1433
q5	4602	4466	4316	4316
q6	223	192	128	128
q7	2072	1960	1793	1793
q8	2634	2491	2719	2491
q9	7453	7560	7493	7493
q10	2993	3156	2658	2658
q11	566	511	470	470
q12	641	761	596	596
q13	3231	3597	3014	3014
q14	272	283	270	270
q15	547	503	489	489
q16	1031	1082	1009	1009
q17	1080	1322	1351	1322
q18	7345	7056	6996	6996
q19	789	715	726	715
q20	1896	1931	1830	1830
q21	4637	4212	4098	4098
q22	1061	1049	993	993
Total cold run time: 52426 ms
Total hot run time: 49987 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

TPC-DS: Total hot run time: 184661 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5251cb76341f0dcbb563308df84579a69c75db90, data reload: false

query1	1055	416	392	392
query2	6569	1583	1567	1567
query3	6753	237	227	227
query4	25480	23160	23076	23076
query5	4781	625	467	467
query6	341	241	220	220
query7	4659	508	307	307
query8	323	268	244	244
query9	8744	2603	2604	2603
query10	499	353	339	339
query11	15282	14894	14750	14750
query12	185	122	116	116
query13	1699	582	452	452
query14	10720	8824	8895	8824
query15	217	202	185	185
query16	7378	677	496	496
query17	1239	815	616	616
query18	2001	415	331	331
query19	214	207	173	173
query20	131	127	121	121
query21	214	134	114	114
query22	3864	3999	3749	3749
query23	32912	31949	31843	31843
query24	8493	2373	2413	2373
query25	613	553	456	456
query26	1244	275	161	161
query27	2794	493	358	358
query28	4333	2144	2133	2133
query29	797	626	519	519
query30	315	254	215	215
query31	801	721	673	673
query32	87	78	71	71
query33	602	382	331	331
query34	817	854	542	542
query35	796	818	735	735
query36	899	926	820	820
query37	124	105	84	84
query38	3288	3329	3235	3235
query39	1489	1409	1391	1391
query40	241	129	119	119
query41	68	63	63	63
query42	128	111	110	110
query43	444	444	413	413
query44	1308	750	762	750
query45	201	191	186	186
query46	855	1002	639	639
query47	1654	1694	1657	1657
query48	402	415	329	329
query49	758	471	418	418
query50	670	682	412	412
query51	3893	3931	3849	3849
query52	117	112	105	105
query53	242	260	194	194
query54	306	305	274	274
query55	101	91	94	91
query56	354	325	354	325
query57	1139	1150	1093	1093
query58	293	271	273	271
query59	2351	2494	2329	2329
query60	356	356	336	336
query61	187	160	160	160
query62	766	809	674	674
query63	233	196	194	194
query64	4554	1176	900	900
query65	4077	3972	3973	3972
query66	1181	431	370	370
query67	15027	14764	14764	14764
query68	4594	971	634	634
query69	540	345	309	309
query70	1245	1223	1135	1135
query71	446	331	312	312
query72	5765	5050	5070	5050
query73	659	571	354	354
query74	8649	8751	8679	8679
query75	3313	3265	2870	2870
query76	3274	1180	698	698
query77	532	418	314	314
query78	9479	9675	8936	8936
query79	1930	912	598	598
query80	1691	688	513	513
query81	557	270	240	240
query82	436	165	129	129
query83	361	265	259	259
query84	270	109	95	95
query85	924	487	446	446
query86	383	283	304	283
query87	3492	3489	3394	3394
query88	2946	2281	2324	2281
query89	391	337	296	296
query90	1769	225	225	225
query91	176	174	145	145
query92	75	73	66	66
query93	1234	1004	657	657
query94	737	437	345	345
query95	496	412	397	397
query96	529	546	289	289
query97	2941	2974	2902	2902
query98	230	222	214	214
query99	1312	1409	1290	1290
Total cold run time: 266034 ms
Total hot run time: 184661 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2025

ClickBench: Total hot run time: 27.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5251cb76341f0dcbb563308df84579a69c75db90, data reload: false

query1	0.06	0.05	0.06
query2	0.11	0.05	0.05
query3	0.26	0.08	0.09
query4	1.60	0.11	0.11
query5	0.28	0.26	0.25
query6	1.16	0.65	0.65
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.57	0.52	0.51
query10	0.56	0.55	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.60
query14	0.99	0.98	0.97
query15	0.82	0.79	0.81
query16	0.39	0.40	0.39
query17	0.99	1.02	1.07
query18	0.23	0.22	0.21
query19	1.83	1.90	1.79
query20	0.02	0.02	0.02
query21	15.44	0.27	0.14
query22	4.69	0.06	0.05
query23	16.03	0.26	0.10
query24	2.05	0.32	0.56
query25	0.06	0.05	0.05
query26	0.14	0.14	0.14
query27	0.08	0.08	0.05
query28	4.02	1.20	1.03
query29	12.64	3.82	3.19
query30	0.28	0.13	0.13
query31	2.82	0.60	0.38
query32	3.24	0.54	0.47
query33	3.17	3.12	3.05
query34	16.84	5.20	4.51
query35	4.58	4.53	4.56
query36	0.65	0.49	0.51
query37	0.12	0.06	0.07
query38	0.07	0.04	0.03
query39	0.05	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.04
Total cold run time: 98.18 s
Total hot run time: 27.31 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

FE Regression Coverage Report

Increment line coverage 100.00% (5/5) 🎉
Increment coverage report
Complete coverage report

morningman
morningman approved these changes Nov 28, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 28, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 28, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 28, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 6a20420 into apache:master Nov 28, 2025
27 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 28, 2025
@CalvinKirs
[[fix](rest-s3)Set AWS Request Checksum Calculation only if not set v…
6d89195 
…oid issues on non-S3 storage (#58467)

AWS SDK for Java v2 uses the system property
AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request
payload checksum calculation.


According to the official discussion AWS SDK v2 Discussion
aws/aws-sdk-java-v2#5802


By default, the SDK may automatically calculate checksums when required.


For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums
are calculated only when necessary, providing both compatibility and
performance benefits.


For non-S3 object storage (such as COS, OSS, OBS), using this default
value may cause errors, for example:


`aws-chunked encoding is not supported with the specified
x-amz-content-sha256 value`

because these services do not support AWS-specific chunked encoding or
SHA256 checksum.
@github-actions github-actions bot mentioned this pull request Nov 28, 2025
Merged
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Dec 1, 2025
@CalvinKirs
branch-4.0: [fix](rest-s3)Set AWS Request Checksum Calculation only i…
0afa74e 
…f not set void issues on non-S3 storage (apache#58467)

apache#58467
@CalvinKirs CalvinKirs mentioned this pull request Dec 1, 2025
Merged
morrySnow pushed a commit that referenced this pull request Dec 1, 2025
@github-actions @CalvinKirs
branch-3.1: [fix](rest-s3)Set AWS Request Checksum Calculation only i…
2504c90 
…f not set void issues on non-S3 storage #58467 (#58506)

Cherry-picked from #58467

Co-authored-by: Calvin Kirs <guoqiang@selectdb.com>
yiguolei pushed a commit that referenced this pull request Dec 2, 2025
@CalvinKirs
branch-4.0: [fix](paimon)Support user-defined S3 config prefixes and …
7ce9386 
…unify to HDFS S3A protocol #57116 (#58544)

#57116 
#58467 
#57933
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@CalvinKirs @nagisa-kunhah
[[fix](rest-s3)Set AWS Request Checksum Calculation only if not set v…
fb8a383 
…oid issues on non-S3 storage (apache#58467)

AWS SDK for Java v2 uses the system property
AwsSystemSetting.AWS_REQUEST_CHECKSUM_CALCULATION to control request
payload checksum calculation.


According to the official discussion AWS SDK v2 Discussion
aws/aws-sdk-java-v2#5802


By default, the SDK may automatically calculate checksums when required.


For AWS S3, setting the default value "WHEN_REQUIRED" ensures checksums
are calculated only when necessary, providing both compatibility and
performance benefits.


For non-S3 object storage (such as COS, OSS, OBS), using this default
value may cause errors, for example:


`aws-chunked encoding is not supported with the specified
x-amz-content-sha256 value`

because these services do not support AWS-specific chunked encoding or
SHA256 checksum.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 30, 2026
@CalvinKirs
branch-3.1: [fix](rest-s3)Set AWS Request Checksum Calculation only i…
d6c2eeb 
…f not set, avoid issues on non-S3 storage (apache#5997)

picked from apache#58467
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@zy-kkk zy-kkk zy-kkk approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@CalvinKirs @hello-stephen @doris-robot @morningman @zy-kkk @yiguolei @morrySnow