Problem statement

Moving to cloud from on-premises infrastructure indeed offers numerous benefits, including scalability, flexibility, and can lower costs for companies that they would otherwise spend on physical servers and on the staff needed to manage them. However, while these services can often help cut costs, it’s not always as straightforward as it seems.

Several factors influence these cost. Hidden charges can surprise you if you're not careful. In this project,we’ll see how to optimize cloud costs effectively.

Imagine a developer with IAM access creates an EC2 instance and attaches a volume filled with sensitive information. After taking daily backup snapshots, the developer decides the instance is no longer needed and deletes it, but forgets to remove the volume and snapshots. As a result, AWS continues to charge for those snapshots without the developer realizing it.

Similar to this, there are countless situations where forgotten resources can lead to rising cloud costs. So it’s crucial to identify and eliminate stale resources to optimize cloud spending.

Solution

One effective strategy for optimizing cloud costs is to identify and eliminate stale resources that continue to incur charges.

We’ll use Boto3 to write Lambda functions that interact with the AWS API to identify stale resources and delete them - (here, snapshots)

Then we can set a trigger in CloudWatch to make the Lambda function truly event-driven.

Key Terms

AWS Lambda

AWS Lambda is a serverless compute service that allows you to run code without provisioning or managing servers. By leveraging AWS Lambda, you can automate the cleanup of stale EBS snapshots.

EBS Snapshots

An EBS snapshot is a backup of your data stored on an Amazon Elastic Block Store (EBS) volume. It captures the state of that volume at a specific time, allowing you to restore or create new volumes from it later.Snapshots are incremental, meaning they only save changes made since the last snapshot, which helps save storage space and costs.

AWS charges for these snapshots even if the associated EC2 instance is deleted. Therefore, it’s crucial to identify and remove stale snapshots to optimize costs.

Boto3

Boto3 is the AWS SDK for Python, allowing developers to easily interact with AWS services from their Python applications by making API calls.

#Import boto3
import boto3 

# Create a client to interact with S3
s3 = boto3.client('s3') 

#Make API Calls
response = s3.list_buckets()
print(response['Buckets'])

Client: A low-level interface that provides direct access to AWS service APIs.

Resource: A higher-level interface that abstracts some of the complexities.

# Creates a resource object for S3
s3 = boto3.resource('s3')

#gets a reference to the S3 bucket named 'my-bucket'.
bucket = s3.Bucket('my-bucket')

# iterates over all objects in that S3 bucket and print its key
for obj in bucket.objects.all():
    print(obj.key)

Lambda Function

The Lambda function uses boto3 to interact with the AWS API to automate the identification and deletion of orphaned EBS snapshots, which consume storage resources without serving any operational purpose.

Here’s how it works:

Retrieve All EBS Snapshots :
- First gather a list of all EBS snapshots in the environment.
Get Active EC2 Instance IDs
- Next, fetch the IDs of all active EC2 instances
Identify and Delete Orphaned Snapshots :
- Iterate through each snapshot and check if it is attached to any volume.
- If a snapshot is not attached to a volume or if the attached volume is not linked to a running instance, the function will initiate the deletion process for that snapshot.

Below is the Lambda function used..

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')

    # Get all EBS snapshots
    response = ec2.describe_snapshots(OwnerIds=['self'])

    # Get all active EC2 instance IDs
    instances_response = ec2.describe_instances(Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
    active_instance_ids = set()

    for reservation in instances_response['Reservations']:
        for instance in reservation['Instances']:
            active_instance_ids.add(instance['InstanceId'])

    # Iterate through each snapshot and delete if it's not attached to any volume or the volume is not attached to a running instance
    for snapshot in response['Snapshots']:
        snapshot_id = snapshot['SnapshotId']
        volume_id = snapshot.get('VolumeId')

        if not volume_id:
            # Delete the snapshot if it's not attached to any volume
            ec2.delete_snapshot(SnapshotId=snapshot_id)
            print(f"Deleted EBS snapshot {snapshot_id} as it was not attached to any volume.")
        else:
            # Check if the volume still exists
            try:
                volume_response = ec2.describe_volumes(VolumeIds=[volume_id])
                if not volume_response['Volumes'][0]['Attachments']:
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as it was taken from a volume not attached to any running instance.")
            except ec2.exceptions.ClientError as e:
                if e.response['Error']['Code'] == 'InvalidVolume.NotFound':
                    # The volume associated with the snapshot is not found (it might have been deleted)
                    ec2.delete_snapshot(SnapshotId=snapshot_id)
                    print(f"Deleted EBS snapshot {snapshot_id} as its associated volume was not found.")

Implementation

Focus on the steps, Once you understand the steps, you can write your own Lambda function based on your requirements and run it.

Create an EC2 instance with the default EBS volume (can create a new volume if you wish)
Create a snapshot of the EBS volume you just created.
Create a Lambda Function:
- Go to the AWS Lambda console and create a new Lambda function.
- Choose Python as the runtime.
Upload Your Code to the editor → Deploy the function to update the changes.

Create a Test Event → Run the Test
Test will fail for following reasons :
1. The default execution time is 3 seconds. We should increase this, but keep in mind that a higher execution time will result in higher costs. Let's set it to 10 seconds.
2. The Lambda function needs to interact with EC2 instances and snapshots to retrieve their details. Therefore, we need to attach appropriate policies. and grant permission for Lambda to perform specific actions on EC2 instances and snapshots.

This is where roles come into play : Roles are used to grant permissions to perform specific actions on AWS resources.

Go to the general configuration settings and increase the execution time to 10 seconds.
Go to configuration → Permissions tab → click on default role (red box)

Add permissions by selecting Create Inline Policy.

The following policies should be selected: Describe Instances, Describe Volumes, Describe Snapshots, and Delete Snapshots.

Now that the policies are attached and permissions are granted, the Lambda function is fully configured.

Results

Now, to check, I deleted the EC2 instance, which also deleted the associated EBS volume. Since the volume was deleted, the snapshot became an orphaned snapshot.

Lambda will detect this orphaned snapshot and delete it, saving you costs that you may not have been aware you were incurring.

Note

All resources created should be deleted after implementing this project to avoid incurring charges later.

Additionally, creating a well-structured Lambda function can help optimize your costs further.

Factors of Lambda Influencing Cost :

Memory Allocation: Higher memory settings increase costs.
Execution Duration: Longer execution times lead to higher charges.
Number of Requests: More invocations mean higher costs.

Cost Optimization for Lambda :

Function Memory size : Adjust memory allocation to match your function’s needs. For example, reducing memory from 512 MB to 256 MB can save approximately $0.10 per million requests.
Optimize Function Duration: Improve your code to reduce execution time. Reducing execution time from 200 ms to 100 ms for a function using 256 MB memory can save $0.01667 per million invocations.
Use Provisioned Concurrency Wisely: Configure provisioned concurrency only for predictable traffic patterns to avoid unnecessary costs.

Surprise Charges :

Cold Start Latency: Functions with infrequent invocations can impact additional latency costs.
Data Transfer: Charges for data transferred between Lambda and other AWS services or the internet.

Other cost optimisation techniques :

Setting Up Budget Alarms : allows you to set custom cost and usage budgets that alert you when you exceed your thresholds. Follow these steps to set up your budget:
- Log in to the AWS Management Console.
- Navigate to the "Budgets" page under the "Billing & Cost Management" section.
- Click on the "Create budget" button.
- Choose the budget type (Cost or Usage), set the budget amount, and configure the alert settings.
Choosing right type of instances : AWS Cost Explorer provides a comprehensive set of tools for visualizing, understanding, and managing your AWS costs and usage over time.
- Access AWS Cost Explorer from the "Billing & Cost Management" console.
- Select the time range and granularity for your cost and usage data.
- Use the filters and groups to break down costs by service, region, or tags.
- Save custom reports for future reference.
Implement AWS Pricing Calculator : The AWS Pricing Calculator is a handy tool for estimating your monthly bill based on your usage patterns. Follow these steps to utilize the pricing calculator:
- Go to the AWS Pricing Calculator website.
- Select the services you plan to use, adjust the parameters, and enter your estimated usage.
- Review the cost estimate and refine as needed.
- Save or share the estimate for future reference.

Cloud Cost Optimisation using Lambda