How to Update ASG and Send Slack Notifications When Capacity Can't Be Fulfilled

1. Create A Launch Template

Select Any Instance Type & most importantly, don’t select anything in the purchasing option.

2. Create The Lambda Functions :-

a. For Changing The Configuration

#50-50-ondem-spot (You can write your own name)
import boto3
import json

def lambda_handler(event, context):
    autoscaling = boto3.client('autoscaling')

    response = autoscaling.update_auto_scaling_group(
        AutoScalingGroupName='new-asg', #write your asg name
        MixedInstancesPolicy={
            'LaunchTemplate': {
                'LaunchTemplateSpecification': {
                    'LaunchTemplateId': 'lt-03786bc2d7a9d1313',
                    'Version': '$Latest'
                },
                'Overrides': [
                    {'InstanceType': 'c5.large'}, # you can select your 
                    {'InstanceType': 'c4.large'}  # own instance type
                ]
            },
            'InstancesDistribution': {
                'OnDemandPercentageAboveBaseCapacity': 50,
                'SpotAllocationStrategy': 'capacity-optimized'
            }
        }
    )

    return {
        'statusCode': 200,
        'body': json.dumps('ASG MixedInstancesPolicy updated successfully!')
    }

b. for sending the notification to Slack, included with dynamodb Table for cooldown

#asg-slack-notifier-2 (Sample Name of Lambda Function)
import json
import urllib3
import boto3
import re
from datetime import datetime, timedelta, timezone

dynamodb = boto3.client('dynamodb')
http = urllib3.PoolManager()

SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK/URL"
COOLDOWN_MINUTES = 5
DDB_TABLE = "ASGSlackCooldown"

def lambda_handler(event, context):
    for record in event['Records']:
        sns_message = record['Sns']['Message']
        print("Raw SNS Message:", sns_message)

        try:
            message_json = json.loads(sns_message)
            asg_name = message_json.get("AutoScalingGroupName", "N/A")
            description = message_json.get("Description", "No description")
            status_msg = message_json.get("StatusMessage", "No status message")

            # ✅ Trigger only if UnfulfillableCapacity or MaxSpotInstanceCountExceeded present
            if not any(err in status_msg for err in [
                "UnfulfillableCapacity",
                "MaxSpotInstanceCountExceeded",
                "InsufficientInstanceCapacity",
                "capacity not available",
                "could not launch spot instances"
            ]):
                print("Not a Spot capacity error. Skipping.")
                return

            # 🔁 Check cooldown
            response = dynamodb.get_item(
                TableName=DDB_TABLE,
                Key={"asgName": {"S": asg_name}}
            )

            now = datetime.now(timezone.utc)
            cooldown_expired = True

            if 'Item' in response:
                cooldown_time = response['Item'].get('cooldownUntil', {}).get('S')
                if cooldown_time:
                    cooldown_dt = datetime.fromisoformat(cooldown_time)
                    if now < cooldown_dt:
                        print(f"Cooldown active for {asg_name} until {cooldown_dt}")
                        return  # skip sending Slack message

            # 🔍 Extract instance type from StatusMessage (OLD CODE snippet as requested)
            status_lower = status_msg.lower()
            instance_type_match = re.search(r"instance type (\w+\d+\.\w+)", status_lower)
            if not instance_type_match:
                instance_type_match = re.search(r"\b([a-z]+\d+\.\w+)\b", status_lower)
            instance_type = instance_type_match.group(1) if instance_type_match else "Not found"

            # 📨 Send Slack Alert
            slack_msg = {
                "text": (
                    f":mega: *ASG Spot Launch Failure Alert!*\n"
                    f"• *Auto Scaling Group:* `{asg_name}`\n"
                    f"• *Message:* {description}\n"
                    f"• *Failed Instance Type:* `{instance_type}`"
                )
            }

            http.request(
                "POST",
                SLACK_WEBHOOK_URL,
                body=json.dumps(slack_msg).encode("utf-8"),
                headers={"Content-Type": "application/json"}
            )

            # ⏲️ Set cooldown
            cooldown_until = (now + timedelta(minutes=COOLDOWN_MINUTES)).isoformat()
            dynamodb.put_item(
                TableName=DDB_TABLE,
                Item={
                    "asgName": {"S": asg_name},
                    "cooldownUntil": {"S": cooldown_until}
                }
            )

        except Exception as e:
            slack_msg = {"text": f"⚠️ Error in Lambda: {str(e)}"}
            http.request(
                "POST",
                SLACK_WEBHOOK_URL,
                body=json.dumps(slack_msg).encode("utf-8"),
                headers={"Content-Type": "application/json"}
            )

3. Create SNS Notification (Create Topic & Subscription)

4. Create Event Bridge Rule :-

If you want to give a specific condition then do this

We have to select the target of changing the configurations of the instance types. Whenever the failed event occurs, it will automatically trigger a lambda function, and that function will change the percentage of On-Demand to 50%

5. Create DynamoDB Table

• Table name: ASGSlackCooldown

• Partition key: asgName (String)

• No sort key needed

• No need to enable TTL (we’ll handle it manually)

6. Set IAM Permissions

We have to give The Following Permissions to the “Slack Notifier” Lambda Function:-

1. AllowDynamoDB

2. AutoScaling

// AllowDyanmoDB
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem"
            ],
            "Resource": "arn:aws:dynamodb:ap-south-1:222937140961:table/ASGSlackCooldown"
        }
    ]
}

//AutoScaling
{ 
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:UpdateAutoScalingGroup",
                "dynamodb:GetItem",
                "dynamodb:PutItem",
                "logs:*"
            ],
            "Resource": "*" //you can specify the resources also
        }
    ]
}

// AWSLambdaBasicExecutionRole
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:ap-south-1:222937140961:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:ap-south-1:222937140961:log-group:/aws/lambda/asg-slack-notifier-2:*"
            ]
        }
    ]
}

And One On “50-50-on-demand-spot” Lambda Function:-

// update50-50-ondem-spot
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:UpdateAutoScalingGroup",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:DescribeLaunchTemplateVersions",
                "ec2:DescribeLaunchTemplates",
                "ec2:RunInstances"
            ],
            "Resource": "*"
        }
    ]
}

// AWSLambdaBasicExecutionRole
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:ap-south-1:222937140961:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:ap-south-1:222937140961:log-group:/aws/lambda/50-50-ondem-spot:*"
            ]
        }
    ]
}

Now Let’s Create an Auto Scaling Group & Test All This :-

Select the Launch Template, which you have created with the version

Override the Launch Template and do the following changes in step 2

Change the Spot Percent to 100 percent & select the instance type, which is rarely available

Create two notifications, one for changing the configuration & other for sending slack notification

You can see it is just after the ASG created :- Notice the Instance type and Instance Distribution

And see our lambda function works successfully, only one notification of failed, Unfulfillable Capacity, & then the lambda function triggers, changes the configuration & launches three instances of type c5.large

And Slack Notification also comes with the alert