AWS Cost Optimization Using Lambda

Introduction

Introduction

Imagine you’re on a road trip, cruising along in your car. As you drive, your vehicle uses fuel—just like it should. But when you reach your destination, you forget to turn off the engine. Now, your car is just sitting there, burning fuel without actually going anywhere. The longer you leave the engine running, the more fuel gets wasted, even though it’s not helping you get any closer to your destination.

This is similar to what happens with AWS resources like EBS snapshots. When you create a snapshot, it’s like filling up your car’s gas tank for the trip ahead. But if you forget to delete old, unused snapshots, they keep taking up storage and racking up costs, just like your engine is burning fuel for no reason.

AWS cost optimization is a critical consideration for businesses looking to maximize the value they receive from cloud services. Among the various AWS services, Amazon Elastic Block Store (EBS) offers scalable storage for Amazon EC2 instances, often through snapshots. However, these snapshots can accumulate over time, leading to increased storage costs, especially if they are no longer needed.

Just like you wouldn’t leave your car running and waste gas after reaching your destination, you shouldn’t let unused snapshots continue to cost you money. That’s where automation comes in. In this guide, we’ll explore how to reduce storage costs by automating the cleanup of unused EBS snapshots with AWS Lambda. This approach will prevent unnecessary charges associated with idle snapshots and ensure a leaner, cost-effective AWS environment. We'll walk through each step, from setting up the EBS volume and snapshot to configuring Lambda for automated cleanup, making the entire process straightforward and efficient.

You can follow the complete process in this guide, and for the full code and further instructions, refer to the GitHub repository.

Workflow Overview

Our cost-optimization process will involve a series of key steps, as outlined below. Following these steps will help you set up an automated system for managing EBS snapshots effectively.

1. Create an EBS volume that will be used to create snapshots.

2. Take a snapshot of that volume as an example.

3. Create a Lambda function to automate the deletion of unused snapshots.

4. Set up the required IAM roles and policies for the Lambda function to manage snapshots.

5. Test the Lambda function to ensure proper execution.

6. Verify the cleanup process to confirm that unused snapshots are deleted.

By following this workflow, you’ll set up a fully automated snapshot management system that not only reduces costs but also minimizes manual intervention.

Step 1 – Create an EBS Volume (gp2)

The first step in this process is to create an EBS volume, which will serve as the basis for our snapshots. For this demonstration, we'll use a General Purpose SSD (gp2) volume, which is cost-effective and suitable for a wide range of applications.

Steps to Create an EBS Volume

1. Navigate to the EC2 Console in your AWS Management Console.

2. In the left sidebar, select Volumes.

3. Click on Create Volume to start the volume creation process.

4. Choose General Purpose (SSD) as the volume type, which is designated as gp2 for its balanced cost and performance.

5. Specify the required size for the volume based on your application needs.

After completing these steps, your new EBS volume is ready, and we’ll use it in the next step to create a snapshot. Creating a volume first is necessary as it provides the data source for the snapshots, enabling us to test the snapshot cleanup process.

Step 2 – Create a Snapshot of the Volume

Now that we have an EBS volume set up, the next step is to create a snapshot. A snapshot captures the state of the volume at a particular point in time, allowing for data restoration if needed. Creating this snapshot also provides us with an item that can be cleaned up later in our automated Lambda process.

Steps to Create a Snapshot

1. Go to the Snapshots section in the EC2 Console sidebar.

2. Click on Create Snapshot.

3. Select the volume created in Step 1 and give the snapshot a descriptive name (e.g., initial-test-snapshot).

4. Click Create Snapshot to complete the process.

At this point, you've created an EBS snapshot that can be managed by AWS Lambda in subsequent steps. Snapshots accumulate easily if not managed, which is why it’s beneficial to automate their cleanup.

---

Step 3 – Create the Lambda Function for Snapshot Cleanup

The next step is to create a Lambda function to handle the automated deletion of unused snapshots. AWS Lambda will allow us to execute a code script that checks for and removes snapshots that are no longer necessary. This function will run on a schedule that we define later, enabling periodic cleanup.

Steps to Create a Lambda Function

1. Navigate to the Lambda Console in the AWS Management Console.

2. Click on Create Function and select Author from Scratch.

3. Name the function cost-optimization and choose Python 3.x as the runtime.

4. For execution permissions, use the default settings for now. Later, we’ll attach specific permissions to this function to manage EC2 snapshots.

5. Click Create Function to proceed.

Your Lambda function is now set up, and in the following steps, we'll customize it with specific permissions and Python code to handle snapshot deletion.

---

Step 4 – Test the Default Lambda Function

With the Lambda function created, it’s essential to test it to ensure that the setup is working correctly. Running the function in its default state, without any custom code, will confirm that the Lambda environment is configured properly.

Steps to Test the Default Lambda Function

1. Open your Lambda function in the Lambda Console.

2. Click Test to initiate a test run.

3. If prompted, create a test event (any default template will work for this initial test).

4. Run the test and verify that the Lambda function executes without errors.

This test confirms that the function is correctly set up and ready for customization. In the next steps, we will add the necessary permissions and custom code to make the Lambda function perform the snapshot cleanup as desired.

Step 5 – Attach Required Policies to the IAM Role

To allow the Lambda function to interact with EC2 resources, we need to attach specific IAM policies to the Lambda function’s execution role. These permissions will enable the function to read EC2 data and manage EBS snapshots.

Steps to Attach IAM Policies

1. Navigate to the IAM Console and select Roles from the sidebar.

2. Find the IAM role associated with your Lambda function (the name will include the function name you provided earlier).

3. Click on the role name to open its settings, and select Attach Policies.

4. Attach the following policies:

   • ec2:DescribeVolumes – Allows the user to view details about EC2 volumes.

   • ec2:DescribeInstances – Allows the user to view details about EC2 instances.

   • ec2:DeleteSnapshot – Allows the user to delete EC2 snapshots.

Attaching these policies enables the Lambda function to interact with snapshots as needed for the cleanup process. Without these permissions, the function would not be able to delete unused snapshots.

---

Step 6 – Copy-Paste the Python Code for Snapshot Cleanup

The full code is available in the GitHub repository if you'd like to explore further.

With the Lambda function and permissions in place, we now need to add the Python code that performs the snapshot cleanup. This script checks for two main conditions:

• If a snapshot’s associated volume no longer exists, it will delete the snapshot.

• If a snapshot’s volume exists but is unattached to any instance, the snapshot will also be deleted.

Steps to Add Python Code

1. Open the Lambda function you created earlier and navigate to the Function Code section.

2. Delete any default code in the editor.

3. Copy and paste the Python code below into the editor:

import boto3

# Initialize Boto3 client for EC2
ec2_client = boto3.client("ec2")

def lambda_handler(event, context):
    # Fetch all snapshots owned by the account
    snapshots = ec2_client.describe_snapshots(OwnerIds=["self"])["Snapshots"]

    for snapshot in snapshots:
        snapshot_id = snapshot["SnapshotId"]
        volume_id = snapshot["VolumeId"]

        # Check if the volume exists
        try:
            volume = ec2_client.describe_volumes(VolumeIds=[volume_id])["Volumes"][0]
            volume_exists = True
        except IndexError:
            volume_exists = False

        # Determine if the snapshot should be deleted
        if volume_exists:
            attachments = volume.get("Attachments", [])
            if not attachments:
                print(f"Deleting snapshot {snapshot_id} for volume {volume_id} as it's not attached to any instance.")
                delete_snapshot(snapshot_id)
            else:
                print(f"Skipping snapshot {snapshot_id} for volume {volume_id} because it’s attached to an instance.")
        else:
            print(f"Deleting snapshot {snapshot_id} for volume {volume_id} as the volume doesn't exist.")
            delete_snapshot(snapshot_id)

def delete_snapshot(snapshot_id):
    try:
        ec2_client.delete_snapshot(SnapshotId=snapshot_id)
        print(f"Successfully deleted snapshot: {snapshot_id}")
    except Exception as e:
        print(f"Error deleting snapshot {snapshot_id}: {str(e)}")

This code defines the logic for identifying unused snapshots and deleting them based on the conditions described. The delete_snapshot function includes error handling to ensure smooth execution.

---

Step 7 – Deploy, Test, and Run the Lambda Function

With the code added to the Lambda function, the next step is to deploy and test it. This test will ensure that the function operates as expected and deletes any snapshots that meet the specified conditions.

Steps to Deploy and Test the Lambda Function

1. Click Deploy in the Lambda Console to save the changes and activate the code.

2. Click Test to run the function. Confirm that it executes without errors.

3. After testing, navigate to the CloudWatch Logs to view the execution log. This will show any snapshots that were deleted, along with details on skipped snapshots.

Reviewing the logs ensures that the Lambda function is working correctly. If the code is set up properly, you’ll see messages confirming the deletion of unused snapshots.

Step 8 – Verify Snapshot Cleanup

After running the Lambda function, it’s time to verify whether the unused snapshots were deleted as expected. This step will confirm that the automated cleanup process is functioning correctly.

Steps to Verify Snapshot Cleanup

1. Go to the EC2 Console and select Snapshots from the sidebar.

2. Review the list of snapshots. Any snapshots associated with deleted or unattached volumes should have been removed by the Lambda function.

3. For additional verification, you can also review the CloudWatch Logs generated by the Lambda function to see details of the snapshots that were deleted.

If the Lambda function performed as expected, you’ll see that unused snapshots have been removed, reducing unnecessary storage costs.

---

Automate Snapshot Cleanup with CloudWatch Scheduling

Additionaly, to fully automate the snapshot cleanup process, you can schedule the Lambda function to run at regular intervals using AWS CloudWatch. CloudWatch allows you to define a cron schedule to trigger the Lambda function, ensuring that snapshots are consistently cleaned up.

Steps to Set Up CloudWatch Scheduling

1. Go to the CloudWatch Console and select Rules under the Events section.

2. Click on Create Rule to start configuring a new rule.

3. Select Event Source as Schedule and define a cron expression or rate expression. For example, you can set it to run daily, weekly, or at a custom interval.

4. In the Targets section, choose Lambda Function as the target and select the Lambda function you created for snapshot cleanup.

5. Click Create Rule to complete the setup.

With this CloudWatch rule in place, the Lambda function will automatically execute at the specified intervals, keeping your snapshots organized and your AWS costs optimized.

---

Conclusion

Using AWS Lambda to automate the deletion of unused EBS snapshots is a practical approach to reduce AWS storage costs and improve overall account management. By implementing this solution, you ensure that old, unneeded snapshots are consistently removed, saving both time and resources. Automating the snapshot cleanup with CloudWatch provides additional peace of mind, as the process will run at regular intervals without manual intervention.

In this guide, we covered the end-to-end process of setting up and testing a Lambda function for snapshot cleanup, attaching necessary permissions, and configuring CloudWatch for automation. By following these steps, you’ve implemented a cost-optimization solution that contributes to a cleaner, more efficient AWS environment.

For additional customization, consider adjusting the Lambda code or scheduling frequency to match your organization’s specific needs. Feel free to explore the code repository for further enhancements or troubleshooting.

---

FAQs

1. Why is it important to delete unused EBS snapshots?

Unused snapshots contribute to AWS storage costs, which can add up significantly over time. Regularly cleaning up these snapshots helps reduce unnecessary expenses, leading to cost savings.

2. What permissions are required for the Lambda function to manage snapshots?

The Lambda function needs AmazonEC2ReadOnlyAccess to view EC2 resources, and AmazonEC2FullAccess to delete snapshots. These permissions should be attached to the Lambda execution role in IAM.

3. How frequently should the Lambda function be scheduled?

The frequency depends on your snapshot usage. For instance, daily or weekly scheduling is common for environments that accumulate snapshots frequently, while monthly scheduling may suffice for less active setups.

4. Does the Lambda function delete snapshots for attached volumes?

No, the code is designed to only delete snapshots if the associated volume is unattached or deleted. Snapshots for active volumes attached to instances will be preserved.

5. Can the Lambda function be modified to retain snapshots based on age?

Yes, you can add additional conditions to the Lambda code to keep snapshots that are within a specific age range. This allows for greater control over snapshot retention policies.

Explore the complete project on GitHub here: GitHub repository for AWS EBS Snapshot Cleanup.