Quarkus 3 application on AWS Lambda- Part 3 Reducing Lambda cold starts with SnapStart and DynamoDB request priming

Introduction

In the part 1 of our series about how to develop, run and optimize Quarkus web application on AWS Lambda, we demonstrated how to write a sample application which uses the Quarkus framework, AWS Lambda, Amazon API Gateway and Amazon DynamoDB. We also made the first Lambda performance (cold and warm start time) measurements and observed quite a big cold start time.

In the part 2 of the series, we introduced Lambda SnapStart and measured how its enabling reduces the Lambda cold start time by more than 50%. We also clearly observed the impact of the AWS SnapStart Snapshot tiered cache in our measurements.

In this part of our article series, we'll introduce how to apply Lambda SnapStart priming techniques by starting with DynamoDB request priming with the goal to even further improve the performance of our Lambda functions.

Sample application with the activated AWS Lambda SnapStart with using DynamoDB request priming

We'll re-use the same sample application introduced in the part 1 of our series.

Activating Lambda SnapStart is also a prerequisite for this method.

Globals:
  Function:
    Handler: io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest
    CodeUri: target/function.zip
    Runtime: java21
    SnapStart:
     ApplyOn: PublishedVersions    
....

This can be done in the globals section of the Lambda functions, in which case SnapStart applies to all Lambda functions defined in the AWS SAM template, or you can add the 2 lines

SnapStart:
 ApplyOn: PublishedVersions

to activate SnapStart only for the individual Lambda function.

You can read more about the concepts behind the Lambda SnapStart in the part 2.

SnapStart and runtime hooks offer you new possibilities to create your Lambda functions for low startup latency. With the pre-snapshot hook, we can prepare our Java application as much as possible for the first call. We load and initialize as much as possible which our Lambda function needs before the snapshot is created. This technique is known as priming.

In this method I will introduce you to the priming of DynamoDB request, which is implemented in the AmazonDynamoDBPrimingResource class.

@Startup
@ApplicationScoped
public class AmazonDynamoDBPrimingResource implements Resource {

    @Inject
    private ObjectMapper objectMapper;

    @Inject
    private DynamoProductDao productDao;

    @PostConstruct
    public void init () {
        Core.getGlobalContext().register(this);
    }

    @Override
    public void beforeCheckpoint(org.crac.Context<? extends Resource> context) throws Exception {
        productDao.getProduct("0");
    }

   @Override
   public void afterRestore(org.crac.Context<? extends Resource> context) throws Exception {
   }

}

We use Lambda SnapStart CRaC runtime hooks here. To do this, we need to declare the following dependency in pom.xml:

<dependency>
   <groupId>io.github.crac</groupId>
   <artifactId>org-crac</artifactId>
</dependency>

AmazonDynamoDBPrimingResource class is annotated with @ Startup annotation (so that this class/bean is initialized directly when the application is started) and implements org.crac.Resource interface. The class registers itself as a CRaC resource in the init method annotated with @PostConstruct annotation. The priming itself happens in the method where we search for the product with the ID equal to 0 in the DynamoDB table. beforeCheckpoint method is a CRaC runtime hook that is invoked before creating the microVM snapshot. We are not even interested in the result of the call productDao.getProduct("0"), but with this call all required classes are loaded and instantiated and the expensive one-time initialisation of HTTP Client (default is Apache HTTP Client ) and Jackson Marshallers (for the purpose of converting Java objects to JSON and vice versa) is carried out. As this is done during the deployment phase of the Lambda function when SnapStart is activated and before the snapshot is created, the snapshot will already contain all of this. After the fast snapshot restore phase during the Lambda invoke, we will gain a lot in performance in case of cold start by priming this way (see measurements below). We therefore prime the DynamoDB request.

To ensure that only this priming takes effect, please either comment out or remove the _ @Startup_ annotation in the following AmazonAPIGatewayPrimingResource class.

Measurements of cold and warm start times of our application with Lambda SnapStart and DynamoDB request priming

In the following, we will measure the performance of our GetProductByIdFunction Lambda function, which we will trigger by invoking curl -H "X-API-Key: a6ZbcDefQW12BN56WEV318" https://{$API_GATEWAY_URL}/prod/products/1.

The results of the experiment are based on reproducing more than 100 cold starts and about 100,000 warm starts with the Lambda function GetProductByIdFunction (we ask for the already existing product with ID=1 ) for the duration of about 1 hour. We give Lambda function 1024 MB memory, which is a good trade-off between performance and cost. We also use (default) x86 Lambda architecture. For the load tests I used the load test tool hey, but you can use whatever tool you want, like Serverless-artillery or Postman.

We will measure with tiered compilation (which is default in Java 21, we don't need to set anything separately) and compilation option XX:+TieredCompilation -XX:TieredStopAtLevel=1. To use the last option, you have to set it in template.yaml in JAVA_OPTIONS environment variable as follows:

Globals:
  Function:
    Handler: io.quarkus.amazon.lambda.runtime.QuarkusStreamHandler::handleRequest
    ...
    Environment:
      Variables:
        JAVA_TOOL_OPTIONS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"

Please also note the effect of the AWS SnapStart Snapshot tiered cache. This means that in the case of SnapStart activation, we get the largest cold starts during the first measurements. Due to the tiered cache, the subsequent cold starts will have lower values. For more details about the technical implementation of AWS SnapStart and its tiered cache, I refer you to the presentation by Mike Danilov: "AWS Lambda Under the Hood". Therefore, I will present the Lambda performance measurements with SnapStart being activated for all approx. 100 cold start times (labelled as all in the table), but also for the last approx. 70 (labelled as last 70 in the table), so that the effect of Snapshot Tiered Cache becomes visible to you. Depending on how often the respective Lambda function is updated and thus some layers of the cache are invalidated, a Lambda function can experience thousands or tens of thousands of cold starts during its life cycle, so that the first longer lasting cold starts no longer carry much weight.

To show the impact of the SnapStart with DynamoDB request priming, we'll also present the Lambda performance measurements without SnapStart being activated from the part 1 and with SnapStart being activated but without applying the priming techniques as measured in the part 2.

Cold (c) and warm (w) start time with tiered compilation in ms:

Cold (c) and warm (w) start time with -XX:+TieredCompilation -XX:TieredStopAtLevel=1 compilation in ms:

Conclusion

In this part of the series, we introduced how to apply Lambda SnapStart priming techniques by starting with DynamoDB request priming with the goal to even further improve the performance of our Lambda functions. We saw that by doing this kind of priming by writing some additional code we could significantly further reduce the Lambda cold start times compared to simply activating the SnapStart. More over we could significantly reduce the maximal value for the Lambda warm start times by preloading classes (as Java lazily loads classes when they are required for the first time) and doing some preinitialization work (by invoking the method to retrieve the product from the DynamoDB table by its ID) which will only happen once during the first warm execution of the Lambda function.

We also saw that -XX:+TieredCompilation -XX:TieredStopAtLevel=1 java compilation very much outperformed the tiered compilation for this type of priming.

We also clearly observed the impact of the AWS SnapStart Snapshot tiered cache in our measurements.

In the next part of our article series, we'll introduce another Lambda SnapStart priming technique which is API Gateway Request Event priming. We'll then measure the Lambda performance by applying it and compare the results with other already introduced approaches.