Building a Scalable Serverless REST API with Lambda, API Gateway

This blog provides a comprehensive overview of one of the most popular serverless architectural patterns on AWS: a REST API powered by API Gateway, AWS Lambda, and DynamoDB. This pattern is ideal for building scalable, cost-effective CRUD (Create, Read, Update, Delete) APIs without managing any servers.

1. Architecture Overview

The flow is designed for high availability, automatic scaling, and security. A user makes an HTTPS request to a custom domain, which is routed through API Gateway to a Lambda function that executes the business logic and interacts with a DynamoDB table.

Architecture Diagram

graph TD
    subgraph "User"
        U[<i class="fa fa-user"></i> User]
    end

    subgraph "AWS Cloud"
        R53[<i class="fa fa-road"></i> Route 53]
        ACM[<i class="fa fa-shield-alt"></i> ACM Certificate]
        APIGW[<i class="fa fa-door-open"></i> API Gateway]
        Lambda[<i class="fa fa-microchip"></i> AWS Lambda]
        IAM[<i class="fa fa-key"></i> IAM Role & Policy]
        DynamoDB[<i class="fa fa-database"></i> DynamoDB Table]
        CW[<i class="fa fa-chart-line"></i> CloudWatch Logs]
        XRay[<i class="fa fa-search-location"></i> AWS X-Ray]
    end

    U -- "1. HTTPS Request<br>api.yourdomain.com" --> R53
    R53 -- "2. Resolves to" --> APIGW
    APIGW -- "3. Triggers" --> Lambda
    Lambda -- "4. Performs CRUD" --> DynamoDB
    DynamoDB -- "5. Returns data" --> Lambda
    Lambda -- "6. Returns response" --> APIGW
    APIGW -- "7. Sends HTTP Response" --> U
    APIGW -. "Uses for SSL/TLS" .-> ACM
    Lambda -. "Executes with" .-> IAM
    Lambda -. "Writes logs to" .-> CW
    Lambda -. "Sends traces to" .-> XRay

    classDef aws fill:#FF9900,stroke:#333,stroke-width:2px;
    class R53,APIGW,Lambda,DynamoDB,ACM,IAM,CW,XRay aws;

Core Components

• Amazon Route 53: AWS’s DNS service. It maps your custom domain name (e.g., api.yourdomain.com) to the API Gateway endpoint.
• Amazon Certificate Manager (ACM): Provides the free SSL/TLS certificate to enable HTTPS for your custom domain, ensuring data is encrypted in transit.
• Amazon API Gateway: Acts as the “front door” for your API. It handles request routing, validation, throttling, and authorization. It receives HTTP requests and triggers the appropriate Lambda function.
• AWS Lambda: The serverless compute service where your business logic lives. The function executes in response to API Gateway triggers, processing the request and interacting with other services like DynamoDB.
• Amazon DynamoDB: A fully managed NoSQL database that provides single-digit millisecond latency at any scale. It’s perfect for storing and retrieving data for your API.
• AWS IAM (Identity and Access Management): Defines the permissions for your Lambda function. The IAM role ensures the function has the “least privilege” access it needs to interact with DynamoDB and write logs to CloudWatch.
• Amazon S3: An S3 bucket is used to store the application’s deployment artifact (a zip file).
• Amazon CloudWatch: All logs generated by the Lambda function are sent to CloudWatch Logs for monitoring, debugging, and auditing purposes. API Gateway also sends logs to CloudWatch.
• AWS X-Ray: An optional but highly recommended service for tracing and analyzing requests as they travel through your application, helping you debug and identify performance bottlenecks.

2. Infrastructure as Code (IaC) with Terraform

The entire infrastructure for this serverless API can be managed using Terraform, which allows for defining and provisioning infrastructure as code. This approach ensures consistency, repeatability, and version control for the architecture.

Key Terraform Components:

• AWS Lambda: The compute service that runs the application code. The Terraform configuration defines the Lambda function’s runtime, handler, memory, timeout, and environment variables.
• API Gateway: The entry point for the API. Terraform configures the API Gateway, including its stages, custom domain names, and integration with the Lambda function.
• IAM Roles and Policies: To ensure security, Terraform creates fine-grained IAM roles and policies that grant the Lambda function the exact permissions it needs to access other AWS services (like DynamoDB or S3) and nothing more.
• S3 Bucket: An S3 bucket is used to store the application’s deployment artifact (a zip file).

The Terraform code is organized into modules, promoting reusability and maintainability. The infrastructure for each environment (dev, qa, prod) is managed separately, using different variable files (.tfvars) for environment-specific configurations.

3. Application Architecture with Spring Boot

The business logic for the API is implemented as a Spring Boot application. Using a framework like Spring Boot inside a Lambda function might seem counterintuitive due to cold starts, but with optimizations like Provisioned Concurrency, it’s a viable and powerful approach for complex applications.

Core Components:

• Lambda Handler: The entry point for the Lambda function is a class that implements AWS’s RequestHandler interface. This handler’s responsibility is to translate the incoming API Gateway request into a format that the Spring Boot application can understand and then proxy the request to it. The SpringBootLambdaContainerHandler from the aws-serverless-java-container library is used for this purpose.
• Spring Boot Application: A standard Spring Boot application with a web starter.
• Controller: A REST controller (@RestController) that defines the API endpoints (e.g., /people, /people/{id}). It handles incoming HTTP requests, deserializes the request body into DTOs (Data Transfer Objects), and calls the appropriate service methods.
• Service: The service layer (@Service) contains the core business logic. It’s responsible for orchestrating the application’s functionality, such as validating data, calling other services, and interacting with the data access layer.
• Repository/DAO: The data access layer is responsible for interacting with the database (e.g., DynamoDB). It’s typically implemented using Spring Data or a custom DAO.
• Mapper: A mapper (e.g., using MapStruct) is used to convert between DTOs and database entities.

Request Flow within the Lambda:

1. API Gateway triggers the Lambda function, invoking the handleRequest method in the handler class.
2. The handler class uses the SpringBootLambdaContainerHandler to forward a request to the Spring Boot application’s DispatcherServlet.
3. The DispatcherServlet routes the request to the appropriate method in @RestController based on the request’s path and HTTP method.
4. The controller method processes the request, calls the service layer to perform the business logic, and returns a response.
5. The response is then propagated back through the layers, eventually being returned to the API Gateway and then to the client.

4. Visualizing the Request Flow

To better understand how a request travels through the system, let’s look at a couple of sequence diagrams.

High-Level Request Flow

This diagram illustrates the journey of a request from the end-user to the Lambda function and back.

sequenceDiagram
    participant User
    participant Route53 as Amazon Route 53
    participant APIGW as API Gateway
    participant Lambda as AWS Lambda
    participant SpringBoot as Spring Boot App

    User->>+Route53: DNS Query for api.yourdomain.com
    Route53-->>-User: API Gateway endpoint IP
    User->>+APIGW: HTTPS Request (e.g., GET /people/123)
    APIGW->>+Lambda: Trigger event
    Lambda->>+SpringBoot: Proxies request
    SpringBoot-->>-Lambda: Processed response
    Lambda-->>-APIGW: Formatted response
    APIGW-->>-User: HTTPS Response

Internal Application Flow

This diagram shows what happens inside the Lambda function, within the Spring Boot application.

sequenceDiagram
    participant Lambda Handler
    participant SpringBootLambdaContainerHandler
    participant DispatcherServlet
    participant PersonController
    participant PersonService

    Lambda Handler->>+SpringBootLambdaContainerHandler: proxy(request, context)
    SpringBootLambdaContainerHandler->>+DispatcherServlet: Forwards request
    DispatcherServlet->>+PersonController: Routes to @GetMapping("/people/{id}")
    PersonController->>+PersonService: findPersonById(id)
    PersonService-->>-PersonController: Returns Person entity
    PersonController-->>-DispatcherServlet: Returns ResponseEntity<PersonDto>
    DispatcherServlet-->>-SpringBootLambdaContainerHandler: Returns response
    SpringBootLambdaContainerHandler-->>-Lambda Handler: Returns AwsProxyResponse

5. Securing the Serverless API

Security is a critical aspect of any API. In our serverless architecture, we have multiple layers of security.

API Gateway Security Features

API Gateway provides several built-in mechanisms to protect your API:

• Throttling and Usage Plans: You can configure throttling limits to prevent your API from being overwhelmed by too many requests. Usage plans allow you to grant API access to specific clients and manage the request rates and quotas for each client.
• API Keys: API keys are alphanumeric strings that you can distribute to your clients. You can use them with usage plans to track and control how your API is being used. While API keys are good for tracking usage, they should not be used as the primary mechanism for authentication or authorization.

Lambda Authorizers

For robust authentication and authorization, API Gateway offers Lambda authorizers (formerly known as custom authorizers). A Lambda authorizer is a Lambda function that you provide to control access to your API.

When a client makes a request to your API, API Gateway calls your Lambda authorizer function with the request context, which can include headers, query string parameters, and other information. The authorizer function then evaluates the information and returns an IAM policy that either allows or denies the request.

Our Terraform microservice module is designed to support a Lambda authorizer. The lambda_authorizer_function_name variable in the module’s configuration can be set to the name of the Lambda function that will act as the authorizer.

There are two types of Lambda authorizers:

1. Token-based authorizers (TOKEN): The client sends a bearer token (e.g., a JSON Web Token – JWT) in a header. The authorizer function validates the token and returns an IAM policy.
2. Request parameter-based authorizers (REQUEST): The authorizer function receives various request parameters (headers, query strings, etc.) and uses them to determine the caller’s identity.

Using a Lambda authorizer provides a powerful and flexible way to secure your API, allowing you to implement custom authentication and authorization logic (e.g., OAuth, SAML, or your own custom token-based system).

IAM Permissions

Even if a request passes through the API Gateway and the Lambda authorizer, the Lambda function itself has a limited set of permissions defined by its IAM execution role. This is the principle of least privilege in action. The Terraform configuration ensures that the Lambda function has only the permissions it needs to perform its job (e.g., read from or write to a specific DynamoDB table) and nothing more. This minimizes the potential impact of a security breach.

6. Key Considerations

Performance Benchmarking

• Tools: Use tools like k6, Artillery, or JMeter to load-test your API.
• Metrics to Track:
  • Latency (p95, p99): The time it takes for 95% and 99% of requests to complete. This is more important than average latency.
  • Requests Per Second (RPS): How many requests your API can handle.
  • Error Rate: The percentage of failed requests under load.

Cost Analysis

• Free Tier: Highlight the generous AWS Free Tier for each service (e.g., 1 million API Gateway calls, 1 million Lambda requests, and 25 GB of DynamoDB storage per month).
• Pricing Breakdown:
  • API Gateway: Priced per million requests (e.g., ~$1.00 per million for REST APIs), plus data transfer out costs.
  • AWS Lambda: Priced based on the number of requests ($0.20 per million) and the duration of execution (per GB-second). Costs can be affected by:
    • Provisioned Concurrency: If you enable this to eliminate cold starts, you pay for the amount of concurrency and the period you configure it for, which is an additional cost.
  • DynamoDB: Priced based on read/write capacity. You can choose between:
    • On-Demand: Pay-per-request for read and write requests. Ideal for unpredictable workloads.
    • Provisioned Throughput: Specify a number of reads and writes per second. Good for predictable workloads and can be cheaper if traffic is consistent.
  • Data Transfer: A frequently overlooked cost. You pay for data transferred out of AWS services to the internet. Data transfer between services in the same region is often free, but there are exceptions.
  • CloudWatch: Costs for log storage, custom metrics, and alarms can add up, though there is a perpetual free tier.

Cold Start Tips

A “cold start” is the extra latency incurred on the first invocation of a Lambda function that hasn’t been used recently.

• Provisioned Concurrency: The most effective way to eliminate cold starts for critical functions. It keeps a specified number of instances warm, but it costs extra.
• Runtime Choice: Interpreted languages like Python and Node.js generally have faster cold start times than compiled languages like Java or C#.
• Optimize Dependencies: Keep your deployment package small. Only include the libraries you absolutely need.
• Use Lambda Power Tuning: This open-source tool helps you find the optimal memory configuration for your function to balance cost and performance. More memory often means a faster CPU and shorter cold starts.
• Use Modern Java Runtimes and Frameworks: While Java has historically had longer cold start times, modern frameworks and runtimes can significantly mitigate this:
  • GraalVM Native Image: GraalVM is a high-performance JDK that can compile Java bytecode ahead-of-time (AOT) into a self-contained native executable. For Lambda, this results in an incredibly fast startup time and lower memory consumption compared to a traditional JVM.
  • Quarkus: A Kubernetes-native Java stack tailored for GraalVM and OpenJDK HotSpot. It’s designed for fast startup and low memory usage, making it an excellent choice for serverless applications. It achieves this by doing as much processing as possible at build time.
  • Micronaut: Another modern, JVM-based framework that is designed for building microservices and serverless applications. It avoids runtime reflection and dependency injection, shifting that work to compile time. This results in faster startup and reduced memory footprint.
  • AWS Lambda SnapStart for Java: A feature that can improve startup performance for Java functions by up to 10x at no extra cost. It works by initializing your function and taking a snapshot of the memory and disk state, which is then cached. When the function is invoked, it resumes from the snapshot instead of initializing from scratch.

7. Implementation Script (Conceptual)

Here’s a high-level script outlining the steps to build the “People” API, consistent with the Spring Boot architecture described.

Step 1: Create the DynamoDB Table

1. Service: DynamoDB
2. Action: Create a table (e.g., people-table).
3. Primary Key: Define a primary key (e.g., personId of type String).
4. Capacity Mode: Choose “On-demand” for unpredictable workloads.

Step 2: Create the IAM Role and Policy for Lambda

1. Service: IAM
2. Action: Create a new Role for Lambda execution.
3. Create Policy: Attach a custom inline policy that grants access to your DynamoDB table (people-table) and CloudWatch Logs.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "dynamodb:DeleteItem",
                "dynamodb:Scan"
            ],
            "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/people-table"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

Step 3: Create the Lambda Function with Spring Boot

1. Service: AWS Lambda
2. Action: Create a new function.
3. Runtime: Java 17 (or your preferred JVM).
4. Execution Role: Attach the IAM role created in Step 2.
5. Handler: Configure the handler to be com.example.StreamLambdaHandler::handleRequest.
6. Code: The core logic is in the Spring Boot application, which is packaged as a JAR file and uploaded to Lambda. Below is a conceptual look at the controller.

PersonController.java (Conceptual)

@RestController
@RequestMapping("/people")
public class PersonController {

    private final PersonService personService;

    public PersonController(PersonService personService) {
        this.personService = personService;
    }

    @PostMapping
    public ResponseEntity<PersonDto> createPerson(@RequestBody PersonDto personDto) {
        PersonDto createdPerson = personService.createPerson(personDto);
        return new ResponseEntity<>(createdPerson, HttpStatus.CREATED);
    }

    @GetMapping("/{personId}")
    public ResponseEntity<PersonDto> getPersonById(@PathVariable String personId) {
        return personService.getPersonById(personId)
                .map(person -> new ResponseEntity<>(person, HttpStatus.OK))
                .orElse(new ResponseEntity<>(HttpStatus.NOT_FOUND));
    }

    // Add methods for GET (all), PUT, and DELETE
}

Step 4: Configure API Gateway

1. Service: API Gateway
2. Action: Create a new REST API (e.g., “People API”).
3. Create Resources & Methods:
   • Create a resource /people.
   • Add a POST method to /people and integrate it with your Lambda function using Lambda Proxy integration.
   • Create a resource /people/{personId}.
   • Add GET, PUT, and DELETE methods to /people/{personId} and integrate them with the same Lambda function.
4. Enable CORS: Enable CORS on your resources if the API will be called from a browser.
5. Deploy API: Deploy your API to a stage (e.g., v1).

Step 5: Set Up Custom Domain (Optional but Recommended)

1. Service: ACM -> Request a public certificate for api.yourdomain.com.
2. Service: API Gateway -> Custom Domain Names -> Create a new mapping for your API and stage.
3. Service: Route 53 -> Create an ‘A’ record for api.yourdomain.com as an Alias to the API Gateway domain name.