Add If-Match and If-None-Match support for PutObject operations

[klaudworks]

This PR implements If-Match and If-None-Match functionality for PutObject requests. A feature added by AWS last year. It addresses #831 and #732 as well as my own requirements that prevent me from using s3proxy in its current state.

I followed the S3 API specification for conditional headers:

• If-Match: Only perform the operation if the object's ETag matches the given value
• If-None-Match: Only perform the operation if the object's ETag doesn't match the given value
• If-None-Match: *: Only perform the operation if the object doesn't exist

I checked the AWS docs to match the HTTP status codes: https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html. See the old and new behavior in the manual test results table.

Workarounds

Unlike for getBlobs, the jcloud libary was missing the functionality to specify ifMatch and ifNoneMatch for putBlobs. Therefore, I have to check and verify the ETag myself.

Manual Test Results

Automated tests can be seen in the diff. I also tested the changes manually by building and spinning up the s3proxy as a docker container and using an Azure storage account as the backend:

docker build -t s3proxy .
docker run -d \
        -p 8080:8080 \
        -e S3PROXY_AUTHORIZATION=none \
        -e S3PROXY_ENDPOINT=http://0.0.0.0:8080 \
        -e JCLOUDS_PROVIDER=azureblob \
        -e JCLOUDS_IDENTITY=my-storage-account-name \
        -e JCLOUDS_CREDENTIAL='my-storage-account-key' \
        -e JCLOUDS_ENDPOINT=https://my-storage-account-name.blob.core.windows.net \
        -e JCLOUDS_AZUREBLOB_AUTH=azureKey \
        --name s3proxy \
        s3proxy

I tested both the old and the new code manually:

Test Case	Header	Condition	Old Code	New Code
1	If-Match	Wrong ETag	❌ 200 OK	✅ 412 Precondition Failed
2	If-Match	Correct ETag	✅ 200 OK	✅ 200 OK
3	If-None-Match	Matching ETag	❌ 200 OK	✅ 412 Precondition Failed
4	If-None-Match	Different ETag	✅ 200 OK	✅ 200 OK
5	If-None-Match	Wildcard (*) on existing	❌ 200 OK	✅ 412 Precondition Failed
6	If-None-Match	Wildcard (*) on new	✅ 200 OK	✅ 200 OK

[gaul]

This looks good, please address comments so I can merge and I'll run a new release afterwards.

[gaul]

Can you enable some s3-tests, specifically removing @pytest.mark.fails_on_s3proxy for test_put_object_ifmatch_failed, test_put_object_ifmatch_nonexisted_failed? I think then you can remove these duplicated tests.

[klaudworks]

Sure, I didn't even notice the python tests. I removed the Java tests and created a PR gaul/s3-tests#3 that enables the mentioned tests as well as the following two tests which are now covered:

• test_put_object_ifnonmatch_failed
• test_put_object_ifnonmatch_overwrite_existed_failed

The python tests are completing successfully now.

================ 242 passed, 104 skipped, 561 deselected, 3 warnings in 29.08s =================
  py311: OK (32.88=setup[3.52]+cmd[29.37] seconds)
  congratulations :) (32.90 seconds)

[gaul]

Can you add a TODO: that this is an emulated operation and therefore not atomic? Also please add to the README under "S3Proxy emulates the following operations":

• conditional PUT object

The long-term plan is to migrate off of jclouds which lacks support to SDKs that do but in the near-term we could add a BlobStore2 class that calls to the Azure (and someday AWS) SDK then falls back to emulation for jclouds blobstores.

[klaudworks]

✅

[gaul]

Please investigate the s3-tests failures. You can run these locally via src/test/resources/run-s3-tests.sh.

[klaudworks]

The tests worked fine for the transient-nio2 provider and only occured for the azurite storage. The issue was that the AzureBlobStore implementation did not follow jclouds BlobStore interface and threw the undocumented KeyNotFoundException instead of returning null when an object doesn't exist.

The interface states that null should be returned in that case and the transient-nio2 provider implements it properly. Thats why those tests worked.

   /**
    * Retrieves the metadata of a {@code Blob} at location {@code container/name}
    *
    * @param container
    *           container where this exists.
    * @param name
    *           fully qualified name relative to the container.
    * @return null if name isn't present or the blob you intended to receive.
    * @throws ContainerNotFoundException
    *            if the container doesn't exist
    */
   @Nullable
   BlobMetadata blobMetadata(String container, String name);

I fixed the AzureBlobStore implementation to handle that case and match the interface.

FYI when Maven uses Java 24 instead of Java 11 the @autoservice annotation processor breaks and doesn't generate the required service registration files. This leads to the azureblob-sdk provider not being found.

[gaul]

FYI when Maven uses Java 24 instead of Java 11 the https://github.com/autoservice annotation processor breaks and doesn't generate the required service registration files. This leads to the azureblob-sdk provider not being found.

Can you explain how to reproduce these symptoms, preferably in an issue?

[gaul]

Thanks for fixing this! I believe that this only affected internal-only calls like the MPU ones and handleBlobMetadata coincidentally worked due to the top-level handling KeyNotFoundException.

[klaudworks]

Sure, I'll make a note and will reproduce it to create an issue tomorrow. It's quite late here already. Thanks for saving me the trouble to maintain a minio instance on Azure 🙂

[gaul]

Thank you for your contribution @klaudworks!