"Automating EBS Snapshot Cleanup: A Lambda-Based Cost Optimization Solution"

Introduction

In this project, we will develop a Lambda function designed to enhance cost optimization by identifying and removing Elastic Block Store (EBS) snapshots that are no longer associated with active EC2 instances. By regularly scanning the AWS environment, the Lambda function will pinpoint orphaned EBS snapshots, which consume storage resources without serving any operational purpose. Once identified, the Lambda function will initiate the deletion process for these redundant snapshots, thereby freeing up storage space and reducing unnecessary storage costs. This automated solution ensures efficient resource utilization while minimizing overhead, aligning with best practices for cost optimization in AWS environments.

Moving to the cloud from on-premises infrastructure indeed offers numerous benefits, including scalability, flexibility, and agility. However, it's crucial to recognize that cloud resources come with associated costs, and optimizing these expenses is paramount to realizing the full potential of cloud adoption.

As a DevOps engineer or cloud engineer, one of your primary responsibilities is to ensure cost efficiency in cloud environments. This involves continuously monitoring resource usage, identifying inefficiencies, and implementing strategies to optimize costs.

For example, consider a scenario where an EC2 instance is provisioned to host a temporary application. Once the application is no longer needed, the instance is terminated, and its associated volume is automatically deleted. However, any snapshots created from that volume may still exist, leading to unnecessary storage costs over time.

To address this issue, we can leverage automation tools like AWS Lambda to develop a solution that identifies and deletes orphaned snapshots. By writing a Lambda function in Python, we can automate the process of scanning for and removing these redundant snapshots, thereby minimizing storage costs for the organization.

By proactively managing costs and implementing efficient cost optimization strategies, DevOps and cloud engineers play a critical role in maximizing the benefits of cloud computing while ensuring cost-effective operations for their organizations.

LETS REPLICATE THIS NOW!!

Step 1 - Create an EC2 Instance giving all the necessary steps where the volume will be attached.

Step 2 - Create a snapshot and attach the volume of our EC2 instance.

Step 3 - Create a Lambda function with less execution time, because the charging functions include time as one of the parameters with this you can create a Lambda function with reduced execution time, leading to lower costs and improved performance for your serverless applications.

Step 4 - Provide the necessary permissions that include snapshots, EC2 instances, and volumes as well for the code to work

Step 5 - Run this below Python code in the Lambda function you created above!

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    # Get all EBS snapshots
    response = ec2.describe_snapshots(OwnerIds=['self'])

    # Get all active EC2 instance IDs
    instances_response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    active_instance_ids = set()

    for reservation in instances_response['Reservations']:
        for instance in reservation['Instances']:
            active_instance_ids.add(instance['InstanceId'])

    # Iterate through each snapshot and delete if it's not attached to any volume or the volume is not attached to a running instance
    for snapshot in response['Snapshots']:
        snapshot_id = snapshot['SnapshotId']
        volume_id = snapshot.get('VolumeId')

        if not volume_id:
            # Delete the snapshot if it's not attached to any volume
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            print(f"Deleted EBS snapshot {snapshot_id} as it was not attached to any volume.")
        else:
            # Check if the volume still exists
            try:
                volume_response = ec2.describe_volumes(VolumeIds=[volume_id])
                if not volume_response['Volumes'][0]['Attachments']:
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as it was taken from a volume not attached to any running instance.")
            except ec2.exceptions.ClientError as e:
                if e.response['Error']['Code'] == 'InvalidVolume.NotFound':
                    # The volume associated with the snapshot is not found (it might have been deleted)
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as its associated volume was not found.")

Step 6 - You would have noticed that the function has worked but still the snapshot hasn't been deleted yet, this is because the function only works when the EC2 instance is not present and the volume too. Delete the EC2 instance and run the function again, this time you'll notice the snapshot has been erased.

Using Cloud watch

Instead of manually testing the Lambda function, you can configure it to be triggered by events using Amazon CloudWatch Events. This approach makes the Lambda function truly event-driven and automates the snapshot cleanup process based on predefined schedules or triggers.

Step 1 - Define the schedule or trigger conditions that will determine when the Lambda function should be invoked. For example, you can schedule the rule to run at regular intervals or trigger it based on specific events such as instance termination.

We came to the end!!

Similarly, you can create Lambda functions for S3 buckets, RDS instances, and EKS clusters, you can automate routine tasks, improve operational efficiency, and ensure consistent management of your AWS resources. This approach aligns with best practices for infrastructure as code and helps maintain a reliable and scalable AWS environment.

I hope you found the information on creating Lambda functions for automating tasks insightful. Thank you for exploring this topic with me, and I look forward to seeing you on the next blog.

Until next time

Happy cloud computing!!!!