The Other Feature of AWS Lambda Provisioned Concurrency — Saving Money

Provisioned concurrency was launched in December 2019 and its primary feature is to enable workloads which require highly consistent latency. It does this by provisioning a configurable amount of AWS Lambda execution environments ready for your function to use at any point. This is different to regular usage where your function will experience a cold start if its concurrency increases from its current value. This functionality and its performance benefits are presented in the AWS Blog Provisioned Concurrency for Lambda Functions.

In this blog post we’ll look at the lesser known benefit of provisioned concurrency, how it also saves you money. Modern day builders have an increasingly long list of technology they need to know about, and yes I’m also suggesting they should learn about the cost implications of their architectures. In most cases builders will be building systems for a business and the thing you build will have a value. Understanding the cost of producing and running that thing is important and will ultimately make you a better builder.

So lets dive in. Lambda is billed on two components, requests and duration. Requests are the number of individual invocations made to your function, however they are called. This could be in response to a API Gateway request, an SQS message, it doesn’t matter. Each invocation no matter the source is billed the same. Duration is how many GB/s are used. Using 1024 MB of memory for 1 second is billed at the same price as using 512 MB of memory for 2 seconds.

In my local region EU-WEST-1 (Ireland) these are the current Lambda prices as of the time of writing.

Regular Billing

Requests $0.20 per 1M requests

Duration $0.0000166667 for every GB-second

Provisioned Concurrency is billed slightly differently, it has 3 components. Requests and duration are the same as regularly billing. Provisioned concurrency also has a set fee for each GB-second you reserve. This is charged whether you use it or not.

Provisioned Concurrency $0.000004646 for every GB-second

Requests $0.20 per 1M requests

Duration $0.0000108407 for every GB-second

These values are so small it’s hard to comprehend, at least for me. But if you add up the two GB-second values of provisioned concurrency it equals about 7% less than the normal billing. However you will only get this saving if your provisioned concurrency period is equal to your function duration. Which means if you’re using the capacity you’ve requested, the billing for that capacity will be less. If you don’t use it, you pay a premium for having it ready and waiting.

So how do I translate this knowledge into saving money for my workload? With most workloads there will be a tipping point at which the reduced cost of provisioned concurrency is countered by periods when provisioned concurrency isn’t fully used. How and when this point is reached will be dependent on the traffic pattern and the amount of concurrency provisioned. The way to skip this complexity is to configure provisioned currency to your lowest level of concurrency.

If your traffic profile has periods of complete inactivity then you will most likely not benefit from this as a cost saving mechanism. If you have a more consistent traffic pattern, it will probably be the case that you benefit more.

For full details of provisioned concurrency read the Lambda pricing page.

A note on premature optimizations. I’ve seen entire startups running very successfully spending less than $100 on Lambda a month. Unless you’re using Lambda to a very high volume the discussion of whether to implement this change could cost more than the savings it brings. Sanity check your workload first. Remember also that if you use this as a cost optimization mechanism, you will also receive the more consistent performance profile for the concurrency you reserve.

In this post I will cover what I consider the best practices for producing useful logs when deploying to AWS Lambda. I’ll explain how AWS Lambda Powertools, an Open-source client library, can help do a lot of the heavy lifting for you. This post is aimed at developers, DevOps engineers and anyone else involved in the production or operation of systems using Lambda.

I started my tech career as a build engineer; I guess we’d call this a junior Ops role now. It was my responsibility to build other people’s code, get it onto the servers and understand if it wasn’t working correctly. You can imagine that logs of every kind became integral to my working life. Ant, Maven, Hudson (now Jenkins), WebLogic, Tomcat, all produced logs and could be a source of information on issues affecting the applications I supported.

Fast forward to today and although the build and continuous integration tooling has changed over time, the biggest difference I’ve seen is with application logs. I used to deploy applications to two servers in production and one server in the dev and test environments. I used to know all the server ip addresses by heart and had scripts to help me with common tasks. Working on a single server most of the time made life simple. If I had a problem, connect to the machine over ssh, grep the application log for ERROR and start debugging. I can’t tell you how many times I went to the first production server, did the same, found nothing and scratched my head, only to then remember I needed to check the second machine. So what does this have to do with AWS Lambda?

Well as a user of Lambda I have no idea what server my code is running on or what is happening with my application logs. It turns out both of these are huge advantages. The placement of my code onto a server spread across multiple availability zones, load balanced automatically and health checked is all work I no longer have to do. Zipping log files, shipping them to a central location and managing storage is no longer a pain for me.

Lambda automatically integrates with Amazon CloudWatch for a fully managed logging solution which makes working with ephemeral execution environments easy. As an application developer all I have to do is produce my log message to standard out and Lambda does the rest. So in this new world what are the best practices when producing application logs?

Include Lambda context information. The function name, its memory allocation, its version is all useful information to help debug problems. Has a problem started after a new version has been released? Did the function get more expensive after a memory allocation change?

Include AWS Identifiers. As well as the AWS request ID, include the X-Ray trace id if you have tracing enabled. Help yourself join information for multiple sources by including all the IDs you’ll need. This is also true for IDs for external or integration systems.

Structure your logs in JSON. Take advantage of the powerful querying in CloudWatch Insights by structuring your logs into JSON.

Log requests and responses. Where the workload allows, log requests to and responses from your function. This isn’t always possible if you’re working with particularly big requests such as encode images.

Include business identifiers. If the log message is about a customer, include the customer ID. If the log message is about a transaction failure, include the transaction ID.

Runtime configurability. Have the ability to change your logging level and sampling without having to re-release your function code. The ability to enable and disable DEBUG logging when you have a problem is fantastic. Since CloudWatch charges are based on amount ingested as well as storage it is a good cost optimization do be able to control both easily.

This may all sound like a lot of additional work and it can be, but AWS Lambda Powertools supports all of the functionality I listed above and more. Powertools currently supports Java and Python, with more language support coming.

For examples of Java Lambda function handlers and the log messages they produce, you can read the AWS blog post.

By default AWS Lambda creates a number of default metrics about your function. These include errors, throttles, duration, concurrency and many more. These are all metrics which can be produced about your function without knowing anything about what your function does. When you want to create your own custom metrics you have a few choices. In this post I will talk about the various ways you can integrate with Amazon CloudWatch including what I think is best practice for working with Lambda.

Firstly, why would you want to create your own custom metrics? In most cases there is information about your function which is important to the business process it is part of. For example recording that a purchase has been made, a seat has been allocated, a booking cancelled, or that any of these were attempted and failed. As your system grows it’s important to develop useful metrics which can be used to monitor the health of your system and business.

If you’re already using AWS Lambda you’ll most likely be aware of the benefits of managed services. Amazon CloudWatch is a fully managed service which provides many monitoring capabilities, metrics being just one. Developers who are familiar with CloudWatch and who have worked on other compute services might use the CloudWatch PutMetricData API to record a single custom metric.

This will absolutely work, but isn’t ideal for Lambda. Because this API call is synchronous and Lambda is billed in part by the duration of each execution, you will be paying Lambda costs while your PutMetricData API call is being processed. This can be further exacerbated if you are making multiple PutMetricData calls for each function request. This can be simplified by batching your metrics into a single API call.

A better way is to use CloudWatch Embedded Metrics Format (EMF). EMF doesn’t make an API call to CloudWatch, instead it produces a JSON object describing your metric and outputs it to standard out. As you probably know, all logs produced to standard out are shipped to CloudWatch. CloudWatch knows to look for JSON objects conforming to the EMF standard and creates metrics on your behalf. This has the advantage of being entirely asynchronous. Low level EMF client libraries are available in Node.js, Python, Java and C#.

The Open-Source Lambda Powertools project has higher level clients which help your interact with EMF more easily. Lambda Powertools are currently available for Java and Python, with more language support on the way.

Before we look at some examples, let’s recap on some of the CloudWatch terms to make sure we understand what the Lambda Powertools functionality is doing. CloudWatch allows you to distinguish between your metrics in a number of ways, the highest level being the namespace. Each namespace can have multiple dimensions.

In the diagram an example is given of an ‘e-commerce-app’ being the namespace and highest level item. Then three dimensions representing other distinct aspects of that functionality.

In this example we use the @Metrics annotations to make this handler Powertools metrics enabled. We can use the annotation variables to set the namespace and a dimension reducing the amount of duplicated code. You can then get access to the low level MetricsLogger using the MetricsUtils Powertools class and record metrics using the putMetric method. Behind the scenes Powertools is validating, serializing and flushing your metrics just before the handler method returns.

Lambda Powertools can help you to record cold starts automatically using another variable within the annotation.

For more information about Powertools visit the documentation pages. You’ll find instructions on how to add it to your project.

Notes on cost.

Using the synchronous PutMetricData API call to CloudWatch is charged at $0.01 per 1000 requests.

EMF is free, but you will incur more cost for ingesting log data, this is charged at $0.50 per GB.

Both methods will incur costs for custom metrics at $0.30 per metric/month.

Detailed pricing information can be found on the CloudWatch pricing page.

One of the most common questions I’m asked as an AWS Solutions Architect is how do I avoid AWS Lambda cold starts, and should I use some mechanism for warming Lambda functions. This post is aimed at developers / DevOps engineers new to Lambda. Hopefully, by the end of it, you’ll understand the Lambda concurrency model and why warming functions are in virtually all cases not required, or of little real benefit.

The Lambda concurrency model is probably very different to what you’re used to if you’ve come from containers, VMs or bare metal. The latter handle multiple threads of execution at any time. Lambda functions operate in a different way. Only one invocation is handled by a single instance of a Lambda function at any one time. A Lambda instance sometimes referred to as an execution environment or a sandbox is a dedicated microVM. These microVMs are incredibly fast to start up and give each owner strong security around where and how their code is being executed.

So each Lambda function can be made up of thousands of microVMs all executing the same code base, working on different requests. A fleet of instances all managed by the Lambda service on your behalf.

It’s the Lambda service’s responsibility to make sure that each invocation of your function is executed as fast as possible. If all the current instances are busy with work, then the Lambda service will instruct a new microVM to start up, copy your code to it and get it ready to handle the waiting request. This process of provisioning a new microVM is called a cold start. When a microVM is not handling a request it isn’t automatically shut down. The Lambda service expects more requests to come, so it keeps instances around for a period of time. If these instances aren’t being kept busy they are shut down. When a request comes to an instance which is idle but ready, this is called a warm start. The instance is up and has all your code ready to go.

Cold starts are slower than warm starts. Although microVMs are quick to start up, each step in the process adds a small amount of time and there will always be a difference between them. In some workloads latency is very important, user-facing websites for example. You want to do everything you can to minimize the effects of cold starts. That is a topic for a very long post of which I’m sure there are many in existence already. There are also a number of posts that recommend mechanisms to warm Lambda functions. This is done by injecting synthetic requests into the Lambda function which aren’t from users, in order for those synthetic requests to hit cold starts and thereafter users to hit warm starts.

Unfortunately, it’s not that simple.

The most common warming mechanisms I see are ones that use CloudWatch scheduled events to trigger a Lambda function every 15 minutes. This will start one Lambda instance, we call this having a concurrency of 1. After the synthetic execution has finished the instance will be available for reuse by any other requests. If a request comes in while the instance is executing another request, then the Lambda service will create a new instance, a cold start and the concurrency will be 2. The warming mechanism hasn’t prevented the second cold start. Simple synthetic traffic generators like this won’t prevent cold starts above the level of concurrency which they themselves generate. Now you absolutely could create a sophisticated synthetic traffic generation mechanism, but it’s going to be, in my opinion, more effort than it’s worth. Lambda handles scaling quickly and effectively so you don’t have to.

The exceptions to this are if you have a very low traffic volume and the value of each execution is high. In this case, it might be worth spending the time and effort into creating a warming mechanism. For the majority of workloads that have unpredictable traffic, warming mechanisms won’t have the effect you want them to have.

At re:Invent 2020 a new packaging format for AWS Lambda functions was introduced; container images. With this functionality came a number of managed base images that customers can use too. They are patched and maintained by AWS and include functionality to simplify running your functions in a container image.

So what is that functionality? It’s an implementation of the Lambda runtime API. This isn’t just for containers, it’s just that with the introduction of this feature, the runtime API has a lot more visibility. You could always use this API to run other languages such as PHP or R.

So let's dive a little deeper and look at what was open-sourced and why it’s important to developers and builders.

Firstly there is the aws-lambda-java-runtime-interface-client (RIC) this is the implementation of the polling client which asks for the next invocation to execute. In a managed environment it knows which handler to execute through the configuration you’ve given. With container packaging, you can specify the handler in the CMD. The RIC uses a native binding to use CURL to make the required API calls. This reduces the latency compared to Java HTTP clients. It also handles all of the weird and wonderful ways in which a customer’s function can error and handles them gracefully.

By open-sourcing this library, AWS has shared a battle-hardened, highly optimized component for others to use. Digging deeper into the RIC, it actually helps even more. Lambda functions are triggered by a vast variety of different event sources, created by different AWS teams. They all produce their events differently and the Lambda service has to handle these. For the Java managed runtime additional work is done to make sure that these events are presented in the best way possible for developers in the form of the aws-lambda-java-events library. Java POJOs for most of the event sources are included. Occasionally there is clean up required to make these event sources match typical Java naming standards. This work is done in the aws-lambda-java-serialization library.

This behind the scenes transform has caused confusion in the past when developers have been writing unit tests for their functions. They would make an event with certain conditions and then fire it through the handler and assert certain reactions. The problems came when the events weren’t transformed by the tests, leading to a serialization error. Open sourcing the serialization library made unit testing much simpler and easier with the introduction of the aws-lambda-java-tests. It includes support for any required serialization of your events and handy ways to load your test events from files.

My hope is that other open-source frameworks such as Quarkus, Micronaut and Spring Native will use these libraries to improve their own Lambda functionality. I also hope that it will drive new innovation in this area.