From Idea to Execution: A Step-by-Step Tutorial on Deploying SQS-Polling Lambda Functions Using CloudFormation

Introduction :-

AWS CloudFormation template is designed for deploying a Lambda function that polls an SQS queue for messages on a regular schedule and takes actions accordingly.

The provided CloudFormation template showcases the deployment of a serverless AWS Lambda function, a fundamental building block for event-driven, scalable applications on the AWS platform. This template is written in YAML format and leverages the AWS Serverless Application Model (AWS SAM) for defining the Lambda function and its related resources.

In this template, you’ll find parameters that provide flexibility for customizing the Lambda function deployment according to specific environment and application requirements. Additionally, it configures security groups, integrates with networking resources within a Virtual Private Cloud (VPC), and sets up alarms for monitoring Lambda function performance.

Parameters :-

  • EnvironmentName: A string parameter for specifying the environment name.

  • ApplicationName: A string parameter for specifying the application name.

  • CodeS3Bucket: A string parameter for specifying the S3 bucket where the Lambda function code artifacts are stored.

  • CodeS3Key: A string parameter for specifying the S3 key for the Lambda function code artifact.

  • NetworkStackName: A string parameter for specifying the name of an existing CloudFormation stack that contains networking resources.

  • SNSStackName: A string parameter for specifying the name of an existing CloudFormation stack that contains SNS resources.

  • SourceSQSAccountId: A string parameter for specifying the AWS account ID from which SQS messages are replicated.

  • VisitsQueueName: A string parameter for specifying the name of an SQS queue.

Resources :-

  • LambdaSecurityGroup: An EC2 Security Group resource used for the Lambda function. It depends on an imported VPC ID from the NetworkStackName.

  • SQSPollerLambdaFunction: The AWS Lambda function resource. It has the following properties:

  • CodeURI: Specifies the location of the Lambda function code.

  • Handler: Specifies the entry point (handler) for the Lambda function.

  • MemorySize: The allocated memory for the Lambda function.

  • Runtime: The runtime environment (Node.js 14.x in this case).

  • Timeout: The maximum execution time for the Lambda function.

  • Policies: Defines permissions for the Lambda function to interact with SQS, SNS, and Secrets Manager.

  • FunctionName: Specifies the name of the Lambda function.

  • Environment: Specifies environment variables for the Lambda function.

  • VpcConfig: Configures the Lambda function to run in a VPC with specific subnet and security group settings.

  • Events: Defines a CloudWatch Events trigger to schedule the Lambda function at a 1-minute interval.

  • LambdaAlarm: A CloudWatch Alarm resource that monitors the Lambda function's error rate and triggers an SNS notification if the error rate exceeds a threshold.

Outputs :-

  • LambdaFunctionConsoleUrl: An output that provides a URL to the Lambda function's console in the AWS Management Console.

This CloudFormation template is designed for deploying a Lambda function that polls an SQS queue for messages on a regular schedule and takes actions accordingly. It is associated with networking resources (VPC) and SNS topics for monitoring and notifications. Additionally, it sets up CloudWatch Alarms to monitor the Lambda function’s error rate.

Deployment Steps

Follow these steps to upload and create the CloudFormation stack using the AWS Management Console:

  1. Sign in to the AWS Management Console: Log in to your AWS account if you haven’t already.

2. Navigate to CloudFormation: Go to the AWS CloudFormation service from the AWS Management Console.

3. Click the “Create stack” button.

4. Upload the CloudFormation template file (YAML).

Yaml file :

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Serverless Lambda function

Parameters:
  EnvironmentName:
    Description: Environment name for the application
    Type: String
    ConstraintDescription: Specify either dev or production
  ApplicationName:
    Description: Application Name
    Type: String
  CodeS3Bucket:
    Description: S3 Bucket in which the artifacts are stored
    Type: String
    MinLength: 3
  CodeS3Key:
    Description: S3 Key for the lambda artifact
    Type: String
    MinLength: 3
  NetworkStackName:
    Description: Name of an active CloudFormation stack of networking resources
    Type: String
    MinLength: 1
    MaxLength: 255
    AllowedPattern: "^[a-zA-Z][-a-zA-Z0-9]*$"
  SNSStackName:
    Description: Name of an active CloudFormation stack of SNS resources
    Type: String
    MinLength: 1
    MaxLength: 255
    AllowedPattern: "^[a-zA-Z][-a-zA-Z0-9]*$"
  SourceSQSAccountId:
    Description: The AWS account from where the SQS messages are to be replicated
    Type: String
    MinLength: 1
  VisitsQueueName:
    Description: The AWS account from where the SQS messages are to be replicated
    Type: String
    MinLength: 1

Mappings: {}

Conditions: {}

Resources:
  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: 
        Fn::ImportValue:
          !Sub ${NetworkStackName}-VpcId
  SQSPollerLambdaFunction:
    Type: AWS::Serverless::Function
    DependsOn: 
    - LambdaSecurityGroup
    Properties:
      CodeUri: 
        Bucket: !Ref CodeS3Bucket
        Key: !Ref CodeS3Key
      Handler: index.sqsPollHander
      MemorySize: 128
      Runtime: nodejs14.x
      Timeout: 30
      Policies:
      - AWSLambdaVPCAccessExecutionRole
      - Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Action:
            - sqs:ChangeMessageVisibility
            - sqs:ChangeMessageVisibilityBatch
            - sqs:DeleteMessage
            - sqs:DeleteMessageBatch
            - sqs:PurgeQueue
            - sqs:ReceiveMessage
          Resource: !Sub arn:aws:sqs:${AWS::Region}:${SourceSQSAccountId}:*
        - Effect: Allow
          Action:
            - sns:Publish
          Resource: !Sub arn:aws:sns:${AWS::Region}:${AWS::AccountId}:*
        - Effect: Allow
          Action:
            - secretsmanager:DescribeSecret
            - secretsmanager:GetSecretValue
            - secretsmanager:ListSecretVersionIds
            - secretsmanager:ListSecrets
          Resource: !Sub arn:aws:secretsmanager:${AWS::Region}:${AWS::AccountId}:secret:${EnvironmentName}-secrets*
      FunctionName: !Sub ${EnvironmentName}-${ApplicationName}
      Environment:
        Variables:
          NODE_ENV: !Ref EnvironmentName
          CARE_EVENT_UPDATES_SNS_TOPIC: 
            Fn::ImportValue: !Sub ${SNSStackName}-CareEventUpdatesSNSTopicArn
          VISITS_QUEUE_URL: !Sub https://sqs.${AWS::Region}.amazonaws.com/${SourceSQSAccountId}/${VisitsQueueName}
      VpcConfig:
        SubnetIds:
          - Fn::ImportValue:
              !Sub ${NetworkStackName}-PrivateSubnet1Id
          - Fn::ImportValue:
              !Sub ${NetworkStackName}-PrivateSubnet2Id
        SecurityGroupIds: 
          - !Ref LambdaSecurityGroup
      Events:
        SQSPollerEventScheduler:
          Type: Schedule
          Properties:
            Schedule: rate(1 minute)
  LambdaAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub ${EnvironmentName}-SQSEventLambdaAlarmForError
      AlarmDescription: Alarm if lambda errors out too many times
      AlarmActions: 
       - { 'Fn::ImportValue': !Sub '${SNSStackName}-MonitoringAlarmsSNSTopicArn' }
      Namespace: AWS/Lambda
      MetricName: Errors
      Dimensions:
      - Name: FunctionName
        Value: !Ref SQSPollerLambdaFunction
      Statistic: Average
      ComparisonOperator: GreaterThanThreshold
      Threshold: '10'
      EvaluationPeriods: '5'
      Period: '60'
    DependsOn: 
    - SQSPollerLambdaFunction

Outputs:
  LambdaFunctionConsoleUrl:
    Description: Console URL for the Lambda Function.
    Value: !Join
      - ''
      - - https://
        - !Ref AWS::Region
        - ".console.aws.amazon.com/lambda/home?region="
        - !Ref AWS::Region
        - "#/functions/"
        - !Ref SQSPollerLambdaFunction

5. Specify Stack Details:

  • Enter a Stack name for your deployment.

  • Provide parameter values as needed.

  • Review and acknowledge the capabilities .

  • You can set additional stack options or tags if necessary.

6. Review and Create:

  • Review the stack details and configuration.

  • Click “Create stack” to initiate the deployment.

7. Monitor Stack Creation:

  • The CloudFormation stack creation process will begin.

  • Monitor the stack events in the AWS Management Console.

Conclusion :-

In conclusion, this AWS CloudFormation template offers a structured and automated approach to deploying a serverless AWS Lambda function within your AWS environment. By parameterizing key values like the environment name, application name, and S3 storage location, it provides the flexibility to adapt to various deployment scenarios.

The template’s use of AWS SAM enables a simplified and concise definition of the Lambda function and its dependencies, making it easier to manage and update over time. It also adheres to best practices by configuring security groups, integrating with a Virtual Private Cloud (VPC), and implementing CloudWatch Alarms for proactive monitoring.

0
Subscribe to my newsletter

Read articles from Mahira Technology Private Limited directly inside your inbox. Subscribe to the newsletter, and don't miss out.

Written by

Mahira Technology Private Limited
Mahira Technology Private Limited

A leading tech consulting firm specializing in innovative solutions. Experts in cloud, DevOps, automation, data analytics & more. Trusted technology partner.