How to use an Amazon S3 trigger to invoke a Lambda function

Amazon S3 and AWS Lambda are two powerful services in the AWS ecosystem, and integrating them can lead to the creation of powerful serverless applications. By using Amazon S3 events to trigger Lambda functions, you can perform real-time processing on data as it enters your S3 buckets.

Here's a brief overview of how the integration works and some potential use cases:

Integration Overview:

Setting Up an S3 Bucket: Before integrating, you should have an S3 bucket in place. Amazon S3 is a scalable object storage service, where you can store any amount of data in bucket containers.
Amazon S3 Events: You can configure your S3 bucket to send events when certain actions occur. This includes object creation, deletion, and more.
Lambda Function: AWS Lambda lets you run code without provisioning or managing servers. You write a function in one of the supported languages (e.g., Python, Node.js), and Lambda executes the function in response to events.
S3 Trigger for Lambda: In the AWS Lambda configuration, you can set up a trigger that points to your S3 bucket. Once set, specific events in the S3 bucket (like object creation or deletion) will trigger the Lambda function.
Execution: When the defined event occurs in S3, it captures the event details and sends it as a payload to the connected Lambda function. The Lambda function then processes this event in real-time.

Use Cases:

Image Processing: When a new image is uploaded to an S3 bucket, a Lambda function can be triggered to process the image. This could include resizing, applying filters, or extracting metadata.
Data Transformation: When new data files (CSV, JSON, etc.) are uploaded to S3, trigger a Lambda to transform the data format, cleanse it, or enrich it.
Real-time Analytics: Analyze incoming data immediately upon upload. For example, analyzing log files as they're generated and sent to S3.
Backup and Archiving: Automatically copy new objects to another S3 bucket or send them to AWS Glacier for long-term archiving.
Notification System: Send notifications, for instance, using Amazon SNS, when a new object is added or deleted from the bucket. This could be used for monitoring or alerting purposes.
Machine Learning: Trigger ML models to make predictions based on new data that's uploaded to S3. For instance, a Lambda function might call Amazon SageMaker with new data for predictions.
Content Validation: Check incoming content for compliance or standards. If a file doesn't meet certain criteria, it could be flagged or removed.
Database Updates: When new data is added to S3, trigger a Lambda function to update a database, for instance, Amazon DynamoDB or RDS.
Video Transcoding: Convert video files to different formats or resolutions using AWS Lambda and services like Amazon Elastic Transcoder as soon as they're uploaded to S3.
Audit and Logging: Track actions performed on the objects in the S3 bucket and store logs or send them to monitoring services.

To fully leverage the potential of S3 and Lambda integration, developers often combine other AWS services (e.g., Amazon DynamoDB, Amazon SNS, or AWS Step Functions) to create comprehensive serverless workflows.

Tutorial: Image Resizing with Lambda using Two S3 Buckets

Below is a step-by-step tutorial to resize images using Amazon S3 and AWS Lambda:

Code files are available at https://github.com/Brain2life/aws-cookbook/tree/main/s3-trigger-lambda

Prerequisites:

An AWS account.
AWS CLI installed and configured with necessary permissions or you can use the AWS Management Console.

Steps:

1. Create two S3 Buckets:

Navigate to the Amazon S3 service in the AWS Management Console.
Click on 'Create bucket', provide a unique name for your bucket, and proceed with the default settings. Click 'Create'.
Create a second S3 bucket for storing resized images

2. Create an IAM Role for Lambda:

Navigate to the IAM service.
Go to 'Roles' and click 'Create role'.
Choose 'Lambda' as the service that will use this role and click 'Next: Permissions'.
Attach the AWSLambdaExecute policy (it provides permissions for Lambda to run and for S3 object read/write actions).
Review and create the role by providing a name.

3. Create a Lambda Function:

Navigate to the AWS Lambda service.
Click 'Create function'.
Choose 'Author from scratch'. Provide a function name.
For the runtime, select a suitable version (e.g., Python 3.8).
Under 'Execution role', select 'Use an existing role' and choose the IAM role you created in step 2.
Click 'Create function'.

4. Add the Image Resizing Code:

In the Lambda function dashboard, scroll to the 'Function code' section.
Use the following sample Python code which uses the PIL (Pillow) library:

import boto3
from PIL import Image
import io

s3_client = boto3.client('s3')

def resize_image(image_content):
    with Image.open(io.BytesIO(image_content)) as image:
        image.thumbnail((128, 128))
        buffer = io.BytesIO()
        image.save(buffer, 'JPEG')
        buffer.seek(0)
        return buffer

def lambda_handler(event, context):
    bucket_name = event['Records'][0]['s3']['bucket']['name']
    file_key = event['Records'][0]['s3']['object']['key']

    # Fetch the image from the original-images-bucket
    response = s3_client.get_object(Bucket=bucket_name, Key=file_key)
    image_content = response['Body'].read()

    # Resize the image
    resized_image_buffer = resize_image(image_content)

    # Save the resized image to the resized-images-bucket
    output_key = 'resized_' + file_key
    s3_client.put_object(Bucket='resized-images-566571321054', Key=output_key, Body=resized_image_buffer, ContentType='image/jpeg')

    return {
        'statusCode': 200,
        'body': f"Image {file_key} successfully resized and saved to resized-images-bucket"
    }

Before deploying, we need the Pillow library. To do this:
- Locally, create a new directory.
- Install Pillow and Boto3 SDK:
```
  pip install \
  --platform manylinux2014_x86_64 \
  --target=package \
  --implementation cp \
  --python-version 3.8 \
  --only-binary=:all: --upgrade \
  pillow boto3
```
- Create a .zip file containing your application code and the Pillow and Boto3 libraries.
```
  cd package
  zip -r ../lambda_function.zip .
  cd ..
  zip lambda_function.zip lambda_function.py
```
- Upload this lambda_function.zip to your Lambda function using the AWS Console or AWS CLI.

5. Add an S3 Trigger:

In the Lambda function dashboard, click on '+ Add trigger'.
Choose 'S3' and then select your bucket.
Set the event type to 'PUT' (or 'All object create events').
Configure other settings as desired and click 'Add'.

6. Testing:

Upload an image to your S3 bucket.
After a few moments, check the resized-images S3 bucket. You should see a resized version of the image with the prefix 'resized_'.

Original "minion.jpg" image size properties:

Resized "minion.jpg" image size properties:

This is a basic example of how you can set up image resizing using AWS Lambda and S3. Depending on your needs, you might want to enhance this with error handling, logging, different storage structures, or integration with other services.