Add S3 remote storage for snapshot save, load, and list

[anisaoshafi]

What

Adds support for saving, loading, and listing snapshots in your own AWS S3 bucket. Before this, a snapshot could only go to a local file or a LocalStack platform pod (pod:).

How

Set your AWS credentials or AWS_PROFILE, then use an s3:// path:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
# or interchangeable with export AWS_PROFILE=...

lstk snapshot save my-pod s3://my-bucket/prefix    # save to S3
lstk snapshot load my-pod s3://my-bucket/prefix    # load it back
lstk snapshot list s3://my-bucket/prefix           # list what's in the bucket

Or use a named AWS profile instead of env vars:

lstk snapshot save my-pod s3://my-bucket/prefix --profile myprofile
lstk snapshot load my-pod s3://my-bucket/prefix --profile myprofile
lstk snapshot list s3://my-bucket/prefix --profile myprofile

• The pod name (my-pod) is the snapshot's name in the bucket. Required for load; auto-generated for save if you leave it out.
• Credentials come from AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY (and optional AWS_SESSION_TOKEN), from --profile, or from the profile named by AWS_PROFILE.
• Credentials can't be put in the URL.

Under the hood

• lstk doesn't talk to S3 itself. It passes your credentials to the emulator, and the emulator does the upload and download. No AWS SDK was added to lstk.
• Credentials aren't stored. lstk registers the bucket with a placeholder URL and sends the real values with each command.
• snapshot list s3://... needs a running emulator, since the emulator is what reads the bucket.

[gtsiolis]

Thanks @carole-lavillonniere for adding cf00471, I was just about to add you as reviewer. 😁

[carole-lavillonniere]

@anisaoshafi I pushed this small clean-up cf00471, hope that looks fine to you!

What about reading env var AWS_PROFILE (if flag --profile not set)?

If we compare the aws authentication mechanism with CLI v1, we are regressing because we don't support credential_process, SSO and assuming role. However I am not sure how much they are used, if we want to support them, and if yes, they could come in a subsequent PR. Something for you to discuss with the rest of the team? It would be interesting to see usage data.

[anisaoshafi]

@anisaoshafi I pushed this small clean-up cf00471, hope that looks fine to you!

What about reading env var AWS_PROFILE (if flag --profile not set)?

If we compare the aws authentication mechanism with CLI v1, we are regressing because we don't support credential_process, SSO and assuming role. However I am not sure how much they are used, if we want to support them, and if yes, they could come in a subsequent PR. Something for you to discuss with the rest of the team? It would be interesting to see usage data.

Good point, and quick win. Will address that, thanks 💯

[anisaoshafi]

@anisaoshafi I pushed this small clean-up cf00471, hope that looks fine to you!

What about reading env var AWS_PROFILE (if flag --profile not set)?

If we compare the aws authentication mechanism with CLI v1, we are regressing because we don't support credential_process, SSO and assuming role. However I am not sure how much they are used, if we want to support them, and if yes, they could come in a subsequent PR. Something for you to discuss with the rest of the team? It would be interesting to see usage data.

We decided to simplify the S3 destination support on purpose, discussed in our past planning sessions with product. For now passing the profiles in either of the ways with --profile or via AWS_ACCESS_KEY_ID+ AWS_SECRET_ACCESS_KEY or AWS_PROFILE should be good to go.

[carole-lavillonniere]

@anisaoshafi I pushed this small clean-up cf00471, hope that looks fine to you!
What about reading env var AWS_PROFILE (if flag --profile not set)?
If we compare the aws authentication mechanism with CLI v1, we are regressing because we don't support credential_process, SSO and assuming role. However I am not sure how much they are used, if we want to support them, and if yes, they could come in a subsequent PR. Something for you to discuss with the rest of the team? It would be interesting to see usage data.

We decided to simplify the S3 destination support on purpose, discussed in our past planning sessions with product. For now passing the profiles in either of the ways with --profile or via AWS_ACCESS_KEY_ID+ AWS_SECRET_ACCESS_KEY or AWS_PROFILE should be good to go.

Thanks for sharing the context @anisaoshafi, makes sense!