We are really pleased to announce the release of version 7 of Soto.

Swift Concurrency

We have replaced the internals of Soto. All the SwiftNIO EventLoopFuture based code has been replaced with Swift concurrency. The advantage of this include

• An easier to read and maintain code base
• A cleaner API surface
• Better support for streaming of requests and responses
• Additional features we weren't able to implement previously (like Event streams)

This does come with one major change to the APIs. Along with the removal of the EventLoop based internal code the EventLoopFuture based APIs have been removed. From this point on Soto will only supply Swift concurrency APIs.

Middleware

Middleware has been re-written to support a Middleware stack result builder. Many of the processes Soto applies to a request have been converted into middleware and when we construct our client and services we build a middleware stack of these eg

self.middleware = AWSMiddlewareStack {
    SigningMiddleware(credentialProvider: credentialProvider)
    RetryMiddleware(retryPolicy: retryPolicyFactory.retryPolicy)
    ErrorHandlingMiddleware(options: options)
}

Much of this is internal and isn't visible to the user but if you want to add multiple middleware to your client you will need to use the middleware stack result builder to pass them to the client.

Full details

You can get the full details of all the changes in the release notes.

An alpha for Soto version 7.0 has just been released. This version has some API improvements and considerable internal changes. Below is listed some of the more major changes and how they affect the library and its users.

Swift Concurrency

The library is now completely written using Swift concurrency. The EventLoopFuture based internals are gone along with the EventLoopFuture APIs. This allows us to make full use of all the features of Swift concurrency and also use the new Swift concurrency based APIs from AsyncHTTPClient which have better support for streamed response payloads.

Streamed Payloads

Operations that returned streamed responses, instead of returning the payload by repeated calling a closure now return the streamed payload as an AWSHTTPBody which conforms to a AsyncSequence of ByteBuffers. You can extract the payload using the pattern below or alternatively use the function AWSHTTPBody.collect(upTo:) to collate the stream of buffers into one.

let result = try await S3.getObject(.init(bucket: "MyBucket", key: "name"))
for try await buffer in result.body {
    processBuffer(buffer)
}

Event streams

Event stream APIs have previous been difficult to implement. S3.SelectObjectContent had a custom implementation, but other operations that received an event stream did not work. The move to AsyncSequence based payloads has simplified implementing these and now there is a generic solution for them all. This solution has been tested on S3.SelectObjectContent and Lambda.InvokeWithResponseStream. The stream of events are accessed from an AWSEventStream which conforms to AsyncSequence. At this point in time there is no support for sending event streams.

Request encoding/Response decoding

The solution for encoding of request header, query values etc and decoding of response header values has always been unsatisfactory. When encoding a request's headers Soto would use Mirror to extract the values out of the input shape. When decoding response headers the header values were added to the input to the Codable decoders, which meant the input had to be in a form the values could be added. JSON reponses had to be decoded to a dictionary before header values were added to the dictionary.

Both of these have been improved by passing a container to the encode/decode functions (via userInfo) which holds the request/response details. The encode functions use their request container to update the request headers, query parameters. The decode functions use their response container to extract header values.

These changes give us performance improvements at both the encoding and decoding stages. We aren't using Mirror while encoding, instead we are writing the values directly to the headers/query arrays during the encode(to:) functions. Decoding of JSON files doesn't require us use our custom DictionaryDecoder. Instead we can use the most up to date JSONDecoder which is considerably faster.

Middleware

Previously Soto middleware provided you with a chance to edit the request before it was sent or the response just after receiving it. This has changed structure to allow more flexibility. You are provided with the request, and a closure to call the next middleware and are expected to return either the response from the closure or an edited version of it. Below is an example middleware.

struct MyMiddleware: AWSMiddlewareProtocol {
    func handle(_ request: AWSHTTPRequest, context: AWSMiddlewareContext, next: AWSMiddlewareNextHandler) async throws -> AWSHTTPResponse {
        let request = processRequest(request)
        let response = try await next(request, context)
        return processResponse(response)
    }
}

This middleware is only editing requests and responses but there is nothing stopping you doing something more complex. In actual fact the middleware is flexible enough now that every part of the Soto request/response processing (signing, error handling, retries, endpoint discovery) is now done with middleware.

Middleware Stack

We also use a result builder to build the middleware stack. When you provide middleware to AWSClient or your service it can be either as an individual middleware or a stack of middleware.

let client = AWSClient(
    middleware: AWSMiddlewareStack {
        AWSLoggingMiddleware()
        MyMiddleware()
    }
)

SotoCrypto removed

The library now has platform requirements of macOS v10.15 and iOS v13. Because of this we can now use the swift-crypto package on all platforms and can get rid of the platform check in the Package.swift. This will fix the issues when cross compiling for Linux on macOS where it would not include swift-crypto. Also it means there is one less package to maintain.

Swift 5.6 introduced a new Swift Package Manager feature: build tool plugins. These allows custom tools to be invoked during the build process. They are primarily used for generating Swift source code from another source.

AWS define the APIs for all their services using the Smithy interface design language (IDL). The majority of Soto is generated code built from these model files. The SotoCodeGenerator project is used to build the Soto source files from the AWS model files. Currently with each new release of Soto, the latest model files are copied from the aws-sdk-go-v2 repository and new Soto source code is generated. This creates a lot of code, 1430181 lines at the last count. Along with the model files Soto is a very large repository to download.

Given Soto source code is generated, could we use a SwiftPM build plugin as a means to avoid downloading all this extra source code? The answer to that is yes, with some gotchas. There is a pull request (at the time of writing this it is still unmerged) that adds a build plugin to the SotoCodeGenerator project. The build plugin generates the Soto source code locally from a Smithy model file.

Using plugin

It is possible to use this build plugin, alongside SotoCore (Repository containing the underlying code that runs Soto) to give a project access to an AWS service without including the whole of Soto as a dependency. You can set this up as follows.

SwiftPM build plugins are a Swift 5.6 feature so your Package.swift should indicate it needs Swift 5.6 or later

// swift-tools-version: 5.6

Where previously you would include Soto as a dependency instead include the following

.package(url: "https://github.com/soto-project/soto-codegenerator", from: "0.6.0"),
.package(url: "https://github.com/soto-project/soto-core.git", from: "6.4.0")

And the target you want to add the generated Soto code to should be setup as follows

.target(
    name: "MySotoServices",
    dependencies: [.product(name: "SotoCore", package: "soto-core")],
    plugins: [.plugin(name: "SotoCodeGeneratorPlugin", package: "soto-codegenerator")]
)

You then need a couple of files from the Soto repository.

• The region endpoint definition file endpoints.json
• The model file for the service you want to use. You can find these here.

Copy both of these into the Source folder of the target where you want to generate the Soto Swift code. If you have multiple targets you are generating code in, you can copy endpoints.json to the root of your project instead. This will avoid having multiple copies of the file across multiple targets. Unless you are using a new region endpoint or new functionality from a service you shouldn't need to update these files again.

Finally if your target only includes AWS Smithy model files in it, you need to add a dummy empty Swift file to the folder. Otherwise SwiftPM will warn you have no source files for your target and not build the target. You could add a comment to this file to document where the generated Swift source files are.

Now build your target

swift build

The build plugin will put the generated Swift code in .build/plugins/outputs/<package name>/<target name>/SotoCodeGeneratorPlugin/GeneratedSources and will include it in the list of Swift files for that target.

A sample project using the SotoCodeGenerator build plugin can be found here.

Gotchas

Earlier I mentioned there were some gotchas using the build plugin.

• If you are dependent on extension code in the Soto repository eg S3 multipart upload, STS/Cognito credential providers or DynamoDB Codable support they will not be available. You need to include Soto as a dependency to have access to these.
• VS Code has some issues when working with build plugins. It looks like SourceKit-LSP is interfering with the build process and it fails to link symbols on the first attempt to build a project. Subsequent builds generally work, but not always. Also once you have generated your source code you need to restart the SourceKit-LSP server (either by restarting VS Code, or invoking the Developer: reload window command) to get code completion, go to definition etc to work. Hopefully these issues can be resolved in the near future.

Removing source code from Soto

In theory I could replace all the generated source code in Soto with the build plugin. This isn't something I am ready to do yet. This feature is still experimental. Also build plugins require Swift 5.6 and Soto supports earlier versions of Swift so this is not possible until we drop support for those earlier versions.

Maybe at some point in the future though you might see a version of Soto with almost all of the Swift source code removed from it.

Soto version 6.0.0 has been released. While this release includes a number of major changes many of these are internal and should not require much change from users of Soto. Upgrading to v6 should not be too painful.

Code Generation

Smithy

The big change that v6 brings in is how we generate the AWS service API files. A couple of years back AWS introduced Smithy: a new language for defining services and SDKs. They are now providing models for all of their services using this new language. All new AWS SDKs are using the Smithy models as the source for code generation. I imagine at some point AWS will stop publishing the old json model format. Smithy has a number of advantages, one of the main ones being it has a published specification. This makes source code generation considerably easier. From v6 onwards the Soto AWS service API files are generated from the Smithy files.

Along with the change to the service model format, we have moved the code generator out of the Soto repository into it's own repo: soto-codegenerator.

Given we using a completely new source for service definitions there are some changes which have filtered into the generated files. Three services are no longer generated: SimpleDB, ImportExport, MobileAnalytics. A number of the service names have changed, if the name included "Service" in it before this has been removed and in some places capitalisation of various characters has changed.

Cleanup

With this being a major release we have used this chance to clean up some variable and enum naming during code generation. Enums are now camel case, previously they were all lowercase. Variables that start with an uppercase acronym now capitalise correctly. Previously it would lowercase the first letter of the acronym and the rest would be left uppercase. This led to many variables being prefixed with aWS. These now have the prefix aws.

HTTP Client

The AWSHTTPClient protocol is no longer public. The library now requires that you use AsyncHTTPClient from swift server team. This change has been implemented to reduce the API surface of Soto and to allow us more flexibility in how we integrate the HTTP client with the rest of the project.

Swift Concurrency

Version 5 of Soto included async/await versions of many of the APIs. Version 6 extends this, with a look to the future as well. There are new async versions of the S3 multipart upload/download functions. The big change though is adding Sendable protocol conformance to all the relevant public objects. This means Soto should be ready for Swift 6 when Sendable conformance is to be required for all async code. In general this is an additive change and will not affect the end user but it does impact APIs where the user provides a closure eg AWSPayload.stream. In these cases the closure will now have to be @Sendable.

Changes

Here is a full list of changes in Soto v6.

Major

• Generate service files from Smithy model files.
• Move Code Generator into its own repo soto-codegenerator.
• AWSHTTPClient, AWSHTTPRequest, AWSHTTPResponse are no longer public symbols. User required to use AsyncHTTPClient as an HTTP client.
• Add Sendable Conformance to all relevant objects/protocols. This includes AWSClient, AWSService, CredentialProvider and AWSShape.
• Add support for automatic HTTP checksum calculation (crc32, crc32c, sha1 and sha256) where checksum tests are supported (S3).
• Added AWSBase64Data to store base64 encoded data. This is to replace all instances of Data in AWS service API input/output shapes.
• Move _payloadOptions to AWSShape and rename to _options.
• AWSResponse.headers type is now NIOHTTP1.HTTPHeaders instead of [String: Any].
• AWSPayload.stream has the byteBufferAllocator parameter removed as it is no longer used.
• Added final to all Shapes that are classes.
• S3: Select Object Content code now uses crc32 from SotoCore, instead of importing zlib. Target CSotoZlib has been removed.

Minor

• Add xmlNamespace to AWSServiceConfig.
• Add AWSShape option .checksumRequired which calculates a checksum of the payload and places it in the relevant header.
• Add AWSShape option .md5ChecksumHeader which indicates the shape has a MD5-Content header which will be calculated if the service config has option .calculateMD5 set to true.
• Add Location.hostname which is used for placing content in the hostname.
• Add Location.headerPrefix which is used for placing a dictionary into multiple headers with the keys prefixed by a string. This was part of the S3 middleware but has now been generalised.
• Added async version of AWSClient.shutdown.
• Where operation input struct has deprecated members add an additional init which doesn't include deprecated members and deprecate old init.
• S3: MD5 checksums are no longer automatically calculated (unless required). You can re-enable the automatic calculation of MD5 checksums using the S3 service option .calculateMD5.
• S3: Added async versions of Multipart Upload/download functions
• S3: Send Expect: 100-continue header to cancel large uploads early if AWS know they are going to fail.

Patch

• Remove retry on NIOConnectionError as AsyncHTTPClient does this for us.
• Only retry on HTTPClient.remoteConnectionClosed in debug builds as this could retry non-idempotent calls even when they have been successful.
• Set user-agent header to "Soto/6.0".
• Fix V4 Signing bug where sequential spaces have to be removed from header values when building canonical request.
• S3: S3RequestMiddleware now preserves trailing "/" when reconstructing URL
• S3: Percent encode additional characters in URL paths, to support S3 like services that require this.

The new release of Swift 5.5 comes with the much anticipated Swift concurrency model. The changes include the async/await syntax, structured concurrency, async sequences plus much more. This is an exciting time to be working in Swift. With the release of v5.9.0 of Soto we have added support for some of these new features.

Service API files

First, all the service commands which return an EventLoopFuture of the response from AWS now have an async/await equivalent. If you are running in an async context then the following code from previous versions of Soto

let request = S3.GetObjectRequest(bucket: "my-bucket", key: "my-file")
let responseFuture = s3.getObject(request).whenComplete { result in
    switch result {
    case .success(let response):
        processResponse(response)
    case .failure(let error):
        processError(error)
    }
}

Can now be replaced with

do {
    let request = S3.GetObjectRequest(bucket: "my-bucket", key: "my-file")
    let response = try await s3.getObject(request)
    processResponse(response)
} catch {
    processError(error)
}

The new code is cleaner and easier to read. It looks more like your standard synchronous code and doesn't rely on closures for processing results of asynchronous functions.

Paginators

Swift 5.5 adds the protocol AsyncSequence. This is similar to the protocol Sequence in that you can create an iterator to iterate through all the elements of the AsyncSequence. The exception is to get next element of an AsyncIterator is an async operation. Soto uses these to represent paginators. Each call to next() will get the next page of results from AWS. For an operation that can be paginated you can create an AsyncSequence paginator and then access all the results in a similar way to how you would with a Sequence, as you see below.

var results: [TestObject] = []
let queryRequest = DynamoDB.QueryInput(
    expressionAttributeValues: [":pk": .s("id"), ":sk": .n("2")],
    keyConditionExpression: "id = :id and version >= :version",
    tableName: tableName
)
let paginator = Self.dynamoDB.queryPaginator(queryRequest, type: TestObject.self)
// treat paginator as sequence of results
for try await response in paginator {
    results.append(contentsOf: response.items ?? [])
}

Waiters

Waiters are a relatively new feature of Soto but they also get an async/await API. In a similar manner to the service API commands, we add an equivalent async/await function for every waiter function that returns an EventLoopFuture. Below the code from previous versions of Soto creates a DynamoDB table and then waits for it to be ready using the waiter waitUntilTableExists.

let request = DynamoDB.CreateTableInput(
    attributeDefinitions: [.init(attributeName: "pk", attributeType: .s)],
    keySchema: [.init(attributeName: "pk", keyType: .hash)],
    tableName: "my-table"
)
dynamoDB.createTable(request)
    .flatMap { _ in
        return dynamoDB.waitUntilTableExists(.init(tableName: "my-table"))
    }
    .whenComplete { result in
        ...
    }

can be replaced with

let request = DynamoDB.CreateTableInput(
    attributeDefinitions: [.init(attributeName: "pk", attributeType: .s)],
    keySchema: [.init(attributeName: "pk", keyType: .hash)],
    tableName: "my-table"
)
_ = try await dynamoDB.createTable(request)
try await dynamoDB.waitUntilTableExists(.init(tableName: "my-table"))

In the original code you can see that two commands have been chained together using flatMap. Chaining of EventLoopFutures has been one of the harder things to get right when working with Swift NIO. When there were issues with your code, the compiler did not always produce very helpful error messages. The new async/await code removes the chaining and thus is easier to work with.

Future

Currently the Soto support for the new Swift concurrency is a thin layer that fits on top of the existing APIs. The internals of Soto are still the same Swift NIO code that was there before. This is to ensure the existing EventLoopFuture APIs are still available for users who cannot upgrade to Swift 5.5, or who would still prefer to work with these APIs.

At some point though there will be a version of Soto that replaces the Swift NIO internals with a Swift concurrency implementation. This will be a breaking change and from that point the EventLoopFuture APIs will no longer be available.

With the release of Soto 5.7 we introduce a new feature: waiters. A waiter is a client side abstraction that polls an AWS resource until a desired state is reached. This is a common task with services that create resources asynchronously. Writing the code to continually poll a resource for a state can be error prone. Writing loops in asynchronous code can be particularly difficult. Waiters are provided to take this responsibility away from the client. Below is an example of creating a DynamoDB table, waiting for it to become active and then putting an item in it.

let input = DynamoDB.CreateTableInput(
    attributeDefinitions: [.init(attributeName: "pk", attributeType: .s)],
    keySchema: [.init(attributeName: "pk", keyType: .hash)],
    provisionedThroughput: .init(readCapacityUnits: 1, writeCapacityUnits: 1),
    tableName: "test-table"
)
let futureResult = dynamoDB.createTable(input)
    .flatMap { _ -> EventLoopFuture<Void> in
        dynamoDB.waitUntilTableExists(.init(tableName: "test-table"))
    }
    .flatMap { _ -> EventLoopFuture<DynamoDB.PutItemOutput> in
        dynamoDB.putItem(.init(item: ["pk": .s("test")], tableName: "test-table"))
    }

Most of the code in this sample is setting up the DynamoDB table. The code to wait for for the table to be active has been reduced to the single line dynamoDB.waitUntilTableExists(...).

AWS provide models for waiters for many situations where you might be waiting on a resource state. Examples include waiting for an EC2 instance to be running EC2.waitUntilInstanceRunning, a RDS database to be available RDS.waitUntilDBInstanceAvailable, an IAM Role to exist IAM.waitUntilRoleExists or for a S3 Object to exist S3.waitUntilObjectExists.

We'd like to announce a new Soto Project package. Soto Cognito Authentication Kit provides an easy to use interface to AWS Cognito. Through Soto Cognito Authentication Kit you can create users, authenticate via username and password, verify access and id JWT tokens, refresh these tokens and authenticate using Secure Remote Password. Along with Cognito Identity it can provide AWS credentials for authenticated users, thus allowing them access to AWS resources. The library has supports for running with either a client authenticated with AWS credentials or an unautheticated client.

Soto Cognito Authentication Kit has been around for a year or so already, but it is now an official Soto Project package. You can find it here.

I have previously written an article on generating pre-signed URLs for S3. This article though covers the situation where a signed URL or set of signed headers are required for IAM authentication outside of the AWS service APIs. Below is detailed two different methods for doing this.

1) Using a service object

The first method uses the service object from a Soto service library. This is the easier of the two methods but requires an AWSClient to manage your AWS credentials and that you have a service object with correct signing name. The method uses either AWSService.signURL for signing URLS or AWSService.signHeaders for generating signed headers.

In the example below we have a function for signing requests to a AWS managed Elasticsearch instance. The function uses signHeaders as the signed URL query parameters would confuse the Elasticsearch instance.

func elasticSearchExecute(
    url: URL, 
    method: HTTPMethod, 
    headers: HTTPHeaders, 
    body: ByteBuffer? = nil
) -> EventLoopFuture<HTTPClient.Response> {
    let es = ElasticsearchService(client: awsClient, region: .useast1)
    return es.signHeaders(
        url: url,
        httpMethod: method,
        headers: headers,
        body: body.map { .byteBuffer($0) } ?? .empty
    ).flatMap { signedHeaders in
        let request = try! HTTPClient.Request(
            url: url,
            method: method,
            headers: signedHeaders,
            body: body.map { .byteBuffer($0) }
        )
        return httpClient.execute(request: request, logger: logger)
    }
}

2) Using the Soto signer directly

The second method uses SotoSignerV4 directly. This requires you have your AWS credentials to hand to initialize the AWSSignerV4. We call AWSSignerV4.processURL on the URL before signing it to clean it up. Cleaning up the URL means sorting the query parameters in alphabetical order and percent encoding as AWS requires.

In the example below we are signing a request to an API Gateway REST interface. Please note for API Gateway interfaces the signing name is execute-api.

import SotoSignerV4

func apiGatewayExecute(
    url: URL, 
    method: HTTPMethod, 
    headers: HTTPHeaders, 
    body: ByteBuffer? = nil
) -> EventLoopFuture<HTTPClient.Response> {
    let credentials: Credential = StaticCredential(
        accessKeyId: "_MYACCESSKEY_", 
        secretAccessKey: "_MYSECRETACCESSKEY_"
    )
    let signer = AWSSigner(credentials: credentials, name: "execute-api", region: "us-east-1")
    // clean up URL
    let processedURL = signer.processURL(url: url)!
    let signedHeaders = signer.signHeaders(
        url: processedURL,
        method: method,
        headers: headers,
        body: body.map { .byteBuffer($0) }
    )
    let request = try! HTTPClient.Request(
        url: processedURL,
        method: method,
        headers: signedHeaders,
        body: body.map { .byteBuffer($0) }
    )
    return httpClient.execute(request: request, logger: logger)
}

Both of these examples use the swift-server AsyncHTTPClient but you should be able to use any HTTP client using these methods.

We'd like to announce a new package available from the Soto project. Soto S3 File Transfer has been implemented to ease moving of files between your local file system and S3.

It provides APIs for copying individual files from your local filesystem to S3, S3 back to your file system and from one S3 bucket to another. If files are above a certain size then the transfer manager will use S3 multi-part upload or copy.

The upload to S3, download from S3 and copy paths are also provided for folders. When copying a folder multiple files will be uploaded in parallel. In addition to a simple copy folder operation there are also sync folder operations which will only copy files that are newer in the source folder and if desired delete files that exist in the destination but not the source folder.

Soto S3 File Transfer can be found here.

One of the most common questions we get about Soto is "How do I pre-sign a URL for uploading to S3?". This is a common pattern where a server provides a pre-signed URL to a client. A pre-signed URL allows you to grant temporary access to a single file in an S3 bucket to someone who normally does not have access. The URL is signed with your credentials and it gives anyone the ability to upload to that object in S3 while it is still valid. Be careful, to not provide these too freely.

Uploading to S3

Assuming you already have an AWSClient, the following will generate an EventLoopFuture that will be fulfilled with a pre-signed URL for uploading to S3 .

let s3 = S3(client: awsClient, region: .useast1)
let signedURLFuture = try s3.signURL(
    url: URL(string: "https://<my-bucket>.s3.us-east-1.amazonaws.com/<my-object>")!,
    httpMethod: .PUT,
    expires: .minutes(15)
)

Replace <my-bucket> with the name of your S3 bucket and <my-object> with the name of the object you want to upload. In this example I am assuming your bucket is in the us-east-1 region. You should replace this, both in the S3 initialization and the unsigned URL, with the region your bucket exists in.

The function signURL returns an EventLoopFuture<URL> and not a URL because the AWSClient which manages credential acquisition, required to sign your URL, could still be in the process of acquiring those credentials.

Downloading from S3

To generate a URL that downloads a file from S3 all you need to do is replace the .PUT with a .GET.

let s3 = S3(client: awsClient, region: .useast1)
let signedURLFuture = try s3.signURL(
    url: URL(string: "https://<my-bucket>.s3.us-east-1.amazonaws.com/<my-object>")!,
    httpMethod: .GET,
    expires: .minutes(15)
)

Including header values

If you want to include some headers values with the URL you have to include the headers while signing it and the client will be required to include exactly the same headers when they use the URL. The following will provide a URL that uploads an object with content type set to "application/json" and canned ACL set to "public-read".

let signedURLFuture = try s3.signURL(
    url: URL(string: "https://<my-bucket>.s3.us-east-1.amazonaws.com/<my-object>")!,
    httpMethod: .PUT,
    headers: ["Content-Type": "application/json", "x-amz-acl": "public-read"],
    expires: .minutes(15)
)

You can find a list of possible headers here.