Building an Event-Driven Order Processing System with AWS Lambda, SQS, DynamoDB, EventBridge, and Terraform
DEV Community

Building an Event-Driven Order Processing System with AWS Lambda, SQS, DynamoDB, EventBridge, and Terraform

Building an Event-Driven Order Processing System with AWS Lambda, SQS, DynamoDB, EventBridge, and Terraform

What We Are Building

The application accepts an order through an HTTP API. The flow is:

Client → API Gateway → Order Lambda → DynamoDB (Orders) → SQS → Processor Lambda → DynamoDB (Orders) → EventBridge

Architecture

The final architecture contains the following AWS services:

  • API Gateway
  • AWS Lambda
  • Amazon DynamoDB
  • Amazon SQS
  • Amazon EventBridge
  • Amazon CloudWatch
  • AWS IAM
  • Terraform

We will use two DynamoDB tables:

  • Orders
  • Products

Prerequisites

Before starting, make sure you have:

  • An AWS account
  • AWS CLI
  • Terraform
  • Node.js
  • AWS credentials configured
  • Basic knowledge of Lambda, DynamoDB, and SQS

You can verify Terraform with:

terraform version

And verify AWS authentication with:

aws sts get-caller-identity

Project Structure

The Terraform project looks like this:

terraform/
├── main.tf
├── variables.tf
├── terraform.tfvars
├── sqs.tf
├── iam.tf
├── lambda.tf
├── api_gateway.tf
├── eventbridge.tf
├── cloudwatch.tf
├── outputs.tf
└── lambda/
    ├── order/
    │   └── index.mjs
    └── processor/
        └── index.mjs

Step 1: Configure Terraform

In main.tf:

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.7"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

provider "archive" {
  source = "hashicorp/archive"
  version = "~> 2.7"
}

The AWS provider allows Terraform to communicate with AWS. The archive provider is used to package the Lambda source code into ZIP files.

Step 2: Define Variables

In variables.tf:

variable "aws_region" {
  type        = string
  description = "AWS region"
}

variable "order_table_name" {
  type        = string
  description = "DynamoDB order table name"
}

variable "product_table_name" {
  type        = string
  description = "DynamoDB product table name"
}

Then configure the values in terraform.tfvars:

aws_region = "ap-southeast-1"
order_table_name = "Orders"
product_table_name = "Products"

Using variables makes the Terraform configuration easier to reuse.

Step 3: Create DynamoDB Tables

The first table stores orders. The second table stores products.

module "dynamodb_order_table" {
  source = "terraform-aws-modules/dynamodb-table/aws"

  name     = var.order_table_name
  hash_key  = "OrderId"
  billing_mode = "PAY_PER_REQUEST"
  attributes = [
    {
      name = "OrderId"
      type = "S"
    }
  ]
}

module "dynamodb_product_table" {
  source = "terraform-aws-modules/dynamodb-table/aws"

  name     = var.product_table_name
  hash_key  = "ProductId"
  billing_mode = "PAY_PER_REQUEST"
  attributes = [
    {
      name = "ProductId"
      type = "S"
    }
  ]
}

The Orders table uses OrderId as its partition key. The Products table uses ProductId as its partition key. We use billing_mode = "PAY_PER_REQUEST" because this project does not require us to manage provisioned read and write capacity.

Step 4: Create the SQS Queue

Now we need a queue between the API and the processing Lambda.

resource "aws_sqs_queue" "order_processing_dlq" {
  name            = "order-processing-dlq"
  message_retention_seconds = 1209600
  sqs_managed_sse_enabled = true
}

resource "aws_sqs_queue" "order_processing" {
  name            = "order-processing"
  visibility_timeout_seconds = 60
  message_retention_seconds = 345600
  receive_wait_time_seconds = 10
  sqs_managed_sse_enabled = true
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.order_processing_dlq.arn
    maxReceiveCount = 4
  })
}

There are a few important settings here:

  • visibility_timeout_seconds = 60: When Lambda receives a message, SQS temporarily hides that message from other consumers.
  • message_retention_seconds = 345600: The main queue keeps messages for up to four days.
  • maxReceiveCount = 4: If the message is received four times without being successfully processed, SQS moves it to the DLQ.

Step 5: Create the Order Lambda

The Order Lambda is responsible for accepting the API request.

resource "aws_lambda_function" "order" {
  function_name = "order-service"
  filename      = data.archive_file.order_lambda.output_path
  source_code_hash = data.archive_file.order_lambda.output_base64sha256
  runtime      = "nodejs22.x"
  handler      = "index.handler"
  role         = aws_iam_role.order_lambda.arn
  timeout      = 10
  memory_size  = 256

  environment {
    variables = {
      ORDER_TABLE = module.dynamodb_order_table.dynamodb_table_id
      PRODUCT_TABLE = module.dynamodb_product_table.dynamodb_table_id
      QUEUE_URL = aws_sqs_queue.order_processing.url
    }
  }
}

The Lambda source code uses the AWS SDK for DynamoDB and SQS.

Step 6: Create API Gateway

Now we expose the Lambda through an HTTP API.

resource "aws_api_gateway" "order_api" {
  name        = "order-api"
  description = "Order API"

  provider = aws

  root_resource_id = id

  default_target_lambda_function = aws_lambda_function.order.arn

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_resource" "orders" {
  rest_api_id = aws_api_gateway.order_api.id
  parent_id   = aws_api_gateway.order_api.root_resource_id
  path_part   = "orders"
}

resource "aws_api_gateway_method" "orders_post" {
  rest_api_id = aws_api_gateway.order_api.id
  resource_id = aws_api_gateway_resource.orders.id
  http_method = "POST"
  authorization = "NONE"
  request_parameters = {
    "method.request.querystring.orderId" = false
  }
}

resource "aws_api_gateway_integration" "orders_post" {
  rest_api_id = aws_api_gateway.order_api.id
  resource_id = aws_api_gateway_resource.orders.id
  http_method = aws_api_gateway_method.orders_post.http_method
  integration_http_method = "POST"
  type        = "LAMBDA"
  uri         = "arn:aws:apigateway:${aws_region}:lambda:path/2015-03-31/functions/${aws_lambda_function.order.arn}/invocations"
}

resource "aws_api_gateway_deployment" "order_api" {
  depends_on = [aws_api_gateway_integration.orders_post]
  deployment_id = "order-api-deployment"
  rest_api_id = aws_api_gateway.order_api.id
  stage_name  = "prod"
}

Step 7: Connect SQS to the Processor Lambda

Now we create the second Lambda.

resource "aws_lambda_function" "processor" {
  function_name = "order-processor"
  filename      = data.archive_file.processor_lambda.output_path
  source_code_hash = data.archive_file.processor_lambda.output_base64sha256
  runtime      = "nodejs22.x"
  handler      = "index.handler"
  role         = aws_iam_role.processor_lambda.arn
  timeout      = 30
  memory_size  = 256

  environment {
    variables = {
      ORDER_TABLE = module.dynamodb_order_table.dynamodb_table_id
      EVENT_BUS_NAME = aws_cloudwatch_event_bus.orders.name
    }
  }
}

Then connect SQS to Lambda:

resource "aws_lambda_event_source_mapping" "order_processing" {
  event_source_arn = aws_sqs_queue.order_processing.arn
  function_name = aws_lambda_function.processor.arn
  batch_size = 1
}

This means Lambda automatically polls the SQS queue and invokes the processor when messages are available.

Step 8: Process the Order

The processor first reads the order from DynamoDB.

const existingOrder = await dynamodb.send(new GetItemCommand({
  TableName: ORDER_TABLE,
  Key: {
    OrderId: {
      S: orderId
    }
  }
}));

If the order doesn't exist, we throw an error:

if (!existingOrder.Item) {
  console.error(`Order ${orderId} does not exist`);
  throw new Error(`Order ${orderId} does not exist`);
}

This is important. We want Lambda to fail in this situation. Why? Because SQS uses the Lambda invocation result to determine whether the message was successfully processed. If Lambda throws an error:

  • Lambda failure
  • SQS message becomes available again
  • Lambda retries
  • Repeated failures
  • DLQ

Step 9: Prevent Duplicate Processing

Amazon SQS provides at-least-once delivery. That means the same message can potentially be delivered more than once. So we cannot assume:

  • 1 message = 1 Lambda invocation

Instead, our processor needs to be idempotent. We store the order status in DynamoDB.

await dynamodb.send(new UpdateItemCommand({
  TableName: ORDER_TABLE,
  Key: {
    OrderId: {
      S: orderId
    }
  },
  UpdateExpression: "SET #status = :processing",
  ConditionExpression: "#status = :pending",
  ExpressionAttributeNames: {
    "#status": "Status"
  },
  ExpressionAttributeValues: {
    ":pending": {
      S: "PENDING"
    },
    ":processing": {
      S: "PROCESSING"
    }
  }
}));

The condition is important: Status must currently be PENDING. Only then can the processor claim the order.

Step 10: Handle Duplicate Messages

After the order is already completed, another copy of the message might arrive. We check the current status:

if (currentStatus === "PROCESSING" || currentStatus === "COMPLETED") {
  console.log(`Order ${orderId} has already been processed or is being processed`);
  return;
}

This makes the processor idempotent.

Step 11: Complete the Order

After processing, we update the order:

await dynamodb.send(new UpdateItemCommand({
  TableName: ORDER_TABLE,
  Key: {
    OrderId: {
      S: orderId
    }
  },
  UpdateExpression: "SET #status = :completed",
  ConditionExpression: "#status = :processing",
  ExpressionAttributeNames: {
    "#status": "Status"
  },
  ExpressionAttributeValues: {
    ":processing": {
      S: "PROCESSING"
    },
    ":completed": {
      S: "COMPLETED"
    }
  }
}));

The state transition is:

  • PENDING
  • PROCESSING
  • COMPLETED

The conditional update prevents the order from being completed from an unexpected state.

Step 12: Publish an Event with EventBridge

Once the order is completed, we publish an event.

resource "aws_cloudwatch_event_bus" "orders" {
  name = "order-events"
}

resource "aws_cloudwatch_event_target" "order_completed" {
  event_bus_name = aws_cloudwatch_event_bus.orders.name
  rule = "order-completed"
  target_id = "order-completed"
  arn      = aws_lambda_function.processor.arn
}

resource "aws_cloudwatch_event_rule" "order_completed" {
  name        = "order-completed"
  event_pattern = jsonencode({
    source = ["order-service"]
    detail-type = ["OrderCompleted"]
  })
  event_bus_name = aws_cloudwatch_event_bus.orders.name
}

The resulting event contains information such as:

{
  "source": "order-service",
  "detail-type": "OrderCompleted",
  "detail": {
    "orderId": "b91eb8b4-0c80-4305-9651-518f6347b8dc"
  }
}

This gives us another useful architectural property. The processor does not need to know what happens after an order is completed. Other systems can subscribe to the event later.

Step 13: Configure IAM

Each Lambda gets its own execution role. The Order Lambda only needs permissions required for its job.

resource "aws_iam_role" "order_lambda" {
  name        = "order-lambda-execution-role"
  description = "Execution role for Order Lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_policy" "order_lambda_policy" {
  name        = "order-lambda-policy"
  description = "Policy for Order Lambda"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "dynamodb:GetItem",
          "dynamodb:PutItem"
        ]
        Effect = "Allow"
        Resource = module.dynamodb_order_table.dynamodb_table_arn
      },
      {
        Action = [
          "sqs:SendMessage"
        ]
        Effect = "Allow"
        Resource = aws_sqs_queue.order_processing.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "order_lambda_attach" {
  role       = aws_iam_role.order_lambda.name
  policy_arn = aws_iam_policy.order_lambda_policy.arn
}

The Processor Lambda has different permissions:

resource "aws_iam_role" "processor_lambda" {
  name        = "processor-lambda-execution-role"
  description = "Execution role for Processor Lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_policy" "processor_lambda_policy" {
  name        = "processor-lambda-policy"
  description = "Policy for Processor Lambda"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "dynamodb:GetItem",
          "dynamodb:UpdateItem"
        ]
        Effect = "Allow"
        Resource = module.dynamodb_order_table.dynamodb_table_arn
      },
      {
        Action = [
          "sqs:ReceiveMessage",
          "sqs:DeleteMessage",
          "sqs:GetQueueAttributes"
        ]
        Effect = "Allow"
        Resource = aws_sqs_queue.order_processing.arn
      },
      {
        Action = "events:PutEvents"
        Effect = "Allow"
        Resource = aws_cloudwatch_event_bus.orders.arn
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "processor_lambda_attach" {
  role       = aws_iam_role.processor_lambda.name
  policy_arn = aws_iam_policy.processor_lambda_policy.arn
}

This follows the principle of least privilege. The processor does not need permission to create DynamoDB tables. The Order Lambda does not need permission to publish EventBridge events. Each role gets only the permissions required by that Lambda.

Step 14: Add CloudWatch Logs

Each Lambda gets a dedicated CloudWatch log group.

resource "aws_cloudwatch_log_group" "order_lambda" {
  name              = "/aws/lambda/${aws_lambda_function.order.function_name}"
  retention_in_days = 7
}

resource "aws_cloudwatch_log_group" "processor_lambda" {
  name              = "/aws/lambda/${aws_lambda_function.processor.function_name}"
  retention_in_days = 7
}

This gives us visibility into the application.

Step 15: Initialize Terraform

Now initialize the project:

terraform init

Then validate the configuration:

terraform validate

You should see:

Success! The configuration is valid.

Step 16: Review the Deployment

Before creating anything:

terraform plan

Terraform will show the resources it intends to create.

Step 17: Deploy

Run:

terraform apply

Terraform will create:

  • API Gateway
  • Lambda
  • IAM Roles
  • DynamoDB
  • SQS
  • SQS DLQ
  • EventBridge
  • CloudWatch Log Groups

Once the deployment finishes, Terraform should report the resources created.

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.