S3Proxy

S3Proxy implements the S3 API and proxies requests, enabling several use cases:

• translation from S3 to Google Cloud, Microsoft Azure, OpenStack Swift, and SFTP
• testing without Amazon by using the local filesystem
• extension via middlewares
• embedding into Java applications

Docker Hub hosts a Docker image and has instructions on how to run it.

Reference manifests show how to wire the Docker image into Kubernetes, including health probes, graceful shutdown, and Secret-based credentials. Helm users may prefer the third-party s3proxy-chart.

Users can download releases from GitHub. Developers can build the project by running mvn package which produces a binary at target/s3proxy. S3Proxy requires Java 17 or newer to run.

Configure S3Proxy via a properties file. An example using the local file system as the storage backend with anonymous access:

s3proxy.authorization=none
s3proxy.endpoint=http://127.0.0.1:8080
jclouds.provider=filesystem
jclouds.filesystem.basedir=/tmp/s3proxy

First create the filesystem basedir:

mkdir /tmp/s3proxy

Next run S3Proxy. Linux and Mac OS X users can run the executable jar:

chmod +x s3proxy
s3proxy --properties s3proxy.conf

Windows users must explicitly invoke java:

java -jar s3proxy --properties s3proxy.conf

Finally test by creating a bucket then listing all the buckets:

$ curl --request PUT http://localhost:8080/testbucket

$ curl http://localhost:8080/
<?xml version="1.0" ?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>CustomersName@amazon.com</DisplayName></Owner><Buckets><Bucket><Name>testbucket</Name><CreationDate>2015-08-05T22:16:24.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>

Maven Central hosts S3Proxy artifacts and the wiki has instructions on Java use.

• aws-s3
• azureblob
• filesystem (on-disk storage)
• google-cloud-storage
• openstack-swift (Keystone v3 only)
• sftp (SFTP storage via Apache MINA SSHD)
• transient (in-memory storage)

See the wiki for examples of configurations.

S3Proxy can be configured to assign buckets to different backends with the same credentials. The configuration in the properties file is as follows:

s3proxy.bucket-locator.1=bucket
s3proxy.bucket-locator.2=another-bucket

In addition to the explicit names, glob syntax can be used to configure many buckets for a given backend.

A bucket (or a glob) cannot be assigned to multiple backends.

S3Proxy can modify its behavior based on middlewares:

• bucket aliasing
• bucket prefix scoping
• bucket locator
• eventual consistency modeling
• large object mocking
• latency
• read-only
• regex rename blobs
• sharded backend containers
• storage class override
• user metadata replacer
• no cache override

S3Proxy can listen on HTTPS by setting the secure-endpoint and configuring a keystore. You can read more about how configure S3Proxy for SSL Support in the dedicated wiki page with Docker, Kubernetes or simply Java.

S3Proxy has broad compatibility with the S3 API, however, it does not support:

• ACLs other than private and public-read, including the x-amz-grant-* headers and any grant naming a specific grantee
• BitTorrent hosting
• bucket inventory, analytics, and metrics configuration
• bucket lifecycle configuration
• bucket logging
• bucket notification configuration
• bucket policies
• bucket policy status
• bucket replication
• conditional delete using If-Match, x-amz-if-match-size or x-amz-if-match-last-modified-time
• CORS bucket operations like getting or setting the CORS configuration for a bucket. S3Proxy only supports a static configuration (see below).
• hosting static websites
• object lock, including legal hold and retention
• object ownership controls
• object server-side encryption, see #402
• object tagging
• object versioning on backends other than aws-s3-sdk, see #74
• paginating ListParts with part-number-marker
• public access block
• reading a single part of a multipart object with partNumber, unless the object has only one part
• requester pays buckets
• restoring archived objects
• select object content
• transfer acceleration
• x-amz-expected-bucket-owner

S3Proxy emulates the following operations:

• multi-part upload on the filesystem and transient backends, which store a stub object to carry the metadata
• object and bucket owners, which are always the same synthetic user

Some limitations depend on the storage backend:

Two backends copy a part on the server but fall back to streaming it through S3Proxy in one case each: google-cloud-storage for a range covering less than the whole object, which GCS cannot copy server-side, and azureblob against an endpoint that refuses Put Block From URL, which Azurite does.

Azure mints ETags like 0x8DD3F4A5F0B2C1E, which S3 SDKs decode as hex and abort the request when they cannot -- the AWS SDK for .NET raises ArgumentOutOfRangeException (Parameter 'hex') and the one for Java reports Input is expected to be encoded in multiple of 2 bytes. azureblob therefore reports it under the -1 suffix S3 gives an object assembled from parts, which every client already treats as a value not to verify, and takes the suffix off again when a conditional request names one. The ETag remains opaque either way: it is not the object's MD5 and nothing can check the object against it. Set s3proxy.azureblob.etag=native to report the bare Azure ETag, as releases before 4.0.0 did.

aws-s3, azureblob and google-cloud-storage perform a conditional PUT on the backend. The filesystem and transient backends resolve If-None-Match as they write and emulate If-Match within a single S3Proxy process, so it does not hold against another writer. Where a backend cannot do either, a conditional PUT is refused rather than emulated, since emulating it would mean a read followed by a write -- which answers correctly only when nothing else is writing that key, and a conditional PUT is asked for precisely because something might be. Set s3proxy.aws-s3.conditional-writes=emulated for an S3-compatible endpoint that does not implement conditional writes itself.

S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties file (and corresponding ENV variables for Docker):

s3proxy.cors-allow-origins=https://example\.com https://.+\.example\.com https://example\.cloud
s3proxy.cors-allow-methods=GET PUT
s3proxy.cors-allow-headers=Accept Content-Type
s3proxy.cors-allow-credential=true

CORS cannot be configured per bucket. s3proxy.cors-allow-all=true will accept any origin and header. Actual CORS requests are supported for GET, PUT, POST, HEAD and DELETE methods. Cross-origin sharing is opt-in: a proxy with no CORS configuration, including one built through S3Proxy.Builder without corsRules, shares nothing. Responses that depend on the request Origin carry Vary: Origin so that shared caches do not serve one origin's response to another.

The wiki collects compatibility notes for specific storage backends.

• GitHub issues
• Stack Overflow
• commercial support

• Ceph s3-tests help maintain and improve compatibility with the S3 API
• fake-s3, gofakes3, minio, S3 ninja, and s3rver provide functionality similar to S3Proxy when using the filesystem backend
• GlacierProxy and SwiftProxy provide similar functionality for the Amazon Glacier and OpenStack Swift APIs
• s3mock - Adobe's s3 mock implementation
• s3proxy-chart - Helm chart for deploying S3Proxy
• sbt-s3 runs S3Proxy via the Scala Build Tool
• swift3 provides an S3 middleware for OpenStack Swift
• Zenko provide similar multi-cloud functionality

Copyright (C) 2014-2026 Andrew Gaul

Licensed under the Apache License, Version 2.0