How to Use SNS and SQS to Get S3 Event Notifications

Introduction

• Briefly introduce the importance of event-driven architecture in AWS.

• Explain the common use case: Getting notifications when an S3 event (like file upload) occurs.

• Highlight the role of SNS and SQS in handling event-based workflows.

Why Use SNS and SQS for S3 Event Notifications?

1. SNS (Simple Notification Service):

   • Real-time notification to multiple subscribers (e.g., Email, Lambda, SQS).

   • Fan-out capability – one event can notify multiple targets simultaneously.

2. SQS (Simple Queue Service):

   • Decouples systems and handles message persistence.

   • Ensures delivery and allows retries if the consumer fails.

3. Why Use Both Together:

   • SNS handles fan-out and notification.

   • SQS ensures message durability and allows async processing.

Step-by-Step Guide to Set Up SNS and SQS for S3 Event Notifications

Prerequisites:

• AWS account

• Basic understanding of AWS S3, SNS, and SQS

Step 1: Create an S3 Bucket

• Navigate to the S3 console.

• Create a new bucket (e.g.,my-bucket-id-0935)

• 

Step 2: Create an SNS Topic

• Go to the SNS console.

• Create a new topic (standard).

• Name the topic (e.g., my-sns-topic)

• 

Step 3: Create an SQS Queue

• Go to the SQS console.

• Create a new queue (standard).

• Name the queue (e.g., my-sqs-queue).

• 

Step 4: Policy Configurations

• In SNS and SQS topics, Go to edit section.

• In edit section update the access policies for both SNS and SQS topics.

SQS Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "<SQS TOPIC ARN>",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "<SNS TOPIC ARN>"
        }
      }
    }
  ]
}

SNS Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SNSPublish",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "SNS:Publish",
      "Resource": "<SNS TOPIC ARN>"
    }
  ]
}

Step 5: Subscribe SQS to the SNS Topic

• Go to the SQS topic console.

• Subscribe to Amazon SNS topic (my-sns-topic arn)

Step 6: Subscribe an Email to the SNS Topic

• In SNS, create a new subscription.

• Choose Email as the protocol.

• Confirm the email subscription.

Step 7: Configure S3 to Send Events to SNS

• Go to the S3 console.

• Navigate to bucket settings → Event Notifications.

• Create a new event notification:

  • Event type: PUT (you can choose according to your need)

  • Destination: SNS Topic (my-sns-topic)

How It Works After Setup

1. File Upload Trigger: When a file is uploaded to S3, the event is triggered.

2. SNS Notification: S3 sends the event to the SNS topic.

3. Fan-Out: SNS sends the notification to both:

   • Email → Immediate alert to team.

   • SQS → Queues the message for further processing.

4. Process via SQS: The message sits in the SQS queue until a consumer (like Lambda) processes it.

Example

S3 file upload:

Email Notification:

SQS: Poll for message

Benefits of This Setup

✅ Real-time notifications with SNS.
✅ Durable message handling with SQS.
✅ Scalability – SNS and SQS handle high-volume traffic efficiently.
✅ Flexibility – Multiple services can subscribe to SNS simultaneously.
✅ Decoupling – SQS allows independent processing and failure handling.

Best Practices

• Use Dead Letter Queues (DLQ) for failed SQS messages.

• Add retry mechanisms to handle transient failures.

• Use SNS message filtering to reduce unnecessary notifications.

• Enable encryption on SNS and SQS for enhanced security.

Conclusion

Combining SNS and SQS for S3 event notifications creates a powerful, scalable, and reliable architecture. SNS handles real-time fan-out, while SQS ensures that no message is lost and allows decoupled processing. This setup is ideal for building event-driven applications and improving system resilience.