✨ feat(storage): add an S3-compatible blob backend#605
Merged
Conversation
gaborbernat
added
enhancement
Member
Author
|
Closes #487 |
gaborbernat
force-pushed
the
feat/s3-blob-backend-487
branch
from
July 18, 2026 02:03
42bbea9 to
47610f9
Compare
Merging this PR will not alter performance
Comparing 
Footnotes
|
gaborbernat
force-pushed
the
feat/s3-blob-backend-487
branch
from
July 18, 2026 02:24
47610f9 to
7f9490f
Compare
Blobs so far lived only on the local filesystem, which ties a node's capacity and durability to one disk. Issue tox-dev#487 asks for an S3-compatible backend behind the capability boundary tox-dev#586 established so an operator can keep content in an object store while metadata stays in redb. The backend implements the shared BlobBackend contract over a minimal SigV4-signed REST client built on the reqwest and rustls the workspace already uses, rather than pulling the full AWS SDK. A streamed write stages to a local temp file and hashes as it goes, so the digest is known before anything reaches S3 and readers can still tail an in-progress stage; commit uploads under the digest key with one PUT below the multipart threshold and bounded concurrent parts above it, aborting the upload on failure so parts do not linger. Reads stream ranged GETs and verify against the digest. Capabilities are declared for what the backend actually does: native range, checksum, and delete; emulated create-if-absent via idempotent content-addressed writes; no listing. The blocking maintenance facade stays filesystem-only. Selection moves into the [blob] config table, validated through the backend's own builder. Credentials never enter configuration or logs; they resolve from the standard AWS environment variables at startup, and a missing key fails fast. Backup snapshots the selection but not the secrets. Tests drive the client against an in-process wiremock S3 double covering put, get, range, head, verify, delete, multipart, retries, aborts, and malformed responses, with a known-answer SigV4 vector, so nothing hits a real bucket.
gaborbernat
force-pushed
the
feat/s3-blob-backend-487
branch
from
July 18, 2026 03:11
7f9490f to
3e97aae
Compare
gaborbernat
enabled auto-merge (squash)
July 18, 2026 03:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Blobs currently live only on a node's local filesystem, tying capacity and durability to one disk. Issue #487 asks for an S3-compatible backend behind the capability boundary that #586 introduced, so an operator can keep content in an object store while metadata stays in redb. 📦
The backend implements the shared
BlobBackendcontract over a small SigV4-signed REST client built on thereqwestandrustlsthe workspace already depends on, instead of pulling in the full AWS SDK. Because a blob's key is its sha256, a streamed write stages to a local temp file and hashes as it goes: the digest is known before anything reaches S3, and readers can still tail an in-progress stage. Commit uploads under<prefix>/sha256/<digest>with a singlePUTbelowmultipart_threshold_bytesand bounded concurrent parts above it, aborting the multipart upload on failure so orphaned parts do not linger. Reads stream rangedGETs and verify against the digest. The declared capabilities match what the backend does: native range, checksum, and delete; create-if-absent emulated through idempotent content-addressed writes; and no listing. The blocking maintenance facade stays filesystem-only. 🔒Selection lives in a new
[blob]config table, validated through the backend's own builder so configuration and runtime agree on what is usable. Credentials never enter configuration or logs: they resolve fromAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, and the optionalAWS_SESSION_TOKENat startup, and selecting S3 without them fails fast.peryx backupsnapshots the selection but never the secrets. The filesystem backend remains the default, so existing deployments are unaffected.