How to Use AWS SNS Message Filtering Feature in a Multi-tenant Architecture

AWS SNS (Simple Notification Service) is a Pub/sub communication service that consists of pushing a message by a publisher to a certain number of subscribers. When the publisher sends the message, every single subscriber will receive the same message asynchronously.

In this article, we will learn how to send a specific message to a consumer, using the SNS Message Filtering feature. We will see some of its advantages, based on a concrete example.

Prerequisites

In this article, you'll need:

• A Free Tier AWS Account. All the resources in this article are eligible for the Free Tier offer.

• A basic knowledge of these AWS services: IAM, Lambda, and DynamoDB.

What is AWS SNS?

The main target of SNS is to send a message from a publisher to one or many subscribers simultaneously. For example, you might want to use SNS in your software architecture when a component "A" needs to send the same message to some other components "B", "C", and "D". "A" pushes the message to a topic. If "B", "C", and "D" subscribe to the topic, they can then consume the message asynchronously, as soon as "A" pushes the message. Component "A" is the publisher. The basic usage of AWS SNS can look like below:

In theory, if you need to send different messages to the subscribers via SNS, you would need to have different topics, each dedicated to delivering one type of specific message. Let's suppose you want to deliver a "Message 1" to Components B and C, and a "Message 2" to Component D. The architecture will look like this:

You can deduce that with this architecture, you will have as many topics as message types to send. As a result, you will have more resources to manage as your architecture grows. It will become more error-prone and difficult to maintain. The overall infrastructure cost can increase as well.

Now that we know what SNS is and the potential downsides it can bring in some situations, we will consider a use case and how to use SNS Message Filtering to implement it.

The Use Case

Description

Depending on your architecture, you might need to send different messages to your components. Let's suppose you are developing a multi-tenant application. One advantage of such an architecture is the ability to tailor the application to the specific business cases of your customers. Not only can you develop global features for all your customers, but you can also develop a feature for one specific customer. A component might want to send a message to another component in particular, but not to all the components.

According to what you know now about AWS SNS, you will be tempted to create as many topics as the types of messages you want to send. The architecture schema should look like in the previous section. However, we explained that this architecture has some downsides. Hopefully, SNS has a message-filtering capability that can help to address those limitations.

What is SNS Message Filtering?

AWS SNS Message Filtering is a feature that allows the delivery of only a subset of messages to the subscriber. It is made possible by attaching a filter policy to the topic subscription. The filter policy is a JSON object. You can define the policy to act on the message attributes or the message body. SNS will compare the policy to the message or the attribute. SNS will deliver the message to the subscriber if there is a match. Otherwise, it will not deliver the message to the subscriber.

Let's use that feature to implement the use case we described earlier.

The Implementation

We will not implement a complete multi-tenant app in this article. For the sake of simplicity, here is the scenario you will simulate:

Suppose you have three (3) customers for your SaaS. Each has a DynamoDB table that stores its features. A single SNS topic will deliver the messages. There are two kinds of messages: global feature messages and specific feature messages. All your customers are concerned by the global features. But a specific feature message should be sent to the concerned customer only. Finally, as SNS cannot trigger a DynamoDB table directly, three Lambda functions will process the messages in the SNS topic according to their defined subscription filter policy, and deliver them to the DynamoDB tables.

As an image means a thousand words, here is the architecture as described:

Thanks to the Message Filtering feature, we can use a single SNS topic instead of three. Let's create all these resources via the AWS Console.

1. Create the SNS Topic

The first resource you'll create is the SNS Topic. In the AWS console, navigate to the Amazon Simple Notification Service page. Create a topic with the name FeatureDispatcher. Hit the Next step button.

On the Create Topic page, you can leave all the configuration values at their defaults and click Create Topic.

Congratulations! You have created your SNS topic!

2. Create the DynamoDB tables

Let's create the DynamoDB tables. It's a quite simple task via the AWS Console.

Navigate to the DynamoDB page and click on Tables in the left panel. Then click on the Create Table button.

Create a table called "customer1", with a partition key named "Id" and of type string. Leave all the remaining parameters as defaults.

Repeat the operation to create customer2 and customer3 tables. If everything goes well, you should now have three DynamoDB tables, as below:

3. Create the Lambda Functions

Now, let's create the Lambda Functions as shown in the architecture schema. Their role is to write into the DynamoDB tables.

I will not focus on the function creation, as it is out of the scope of this article. James Eastham has wonderful videos on his YouTube channel to help you achieve that. Nevertheless, let me provide the source code of my lambda function, written in C# .NET 8. The code is nearly identical for the three functions. Only the DynamoDB table name varies. My function.cs file looks like this:

using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DocumentModel;
using Amazon.Lambda.Core;
using Amazon.Lambda.SNSEvents;

namespace CustomerFeatures;

public class Function
{
    /// <summary>
    /// Default constructor. This constructor is used by Lambda to construct the instance. When invoked in a Lambda environment
    /// the AWS credentials will come from the IAM role associated with the function and the AWS region will be set to the
    /// region the Lambda function is executed in.
    /// </summary>

    private readonly AmazonDynamoDBClient dynamoDBClient;
    private readonly string tableName;

    public Function()
    {
        dynamoDBClient = new AmazonDynamoDBClient();
        tableName = "customer1"; // customer2, customer3 ...
    }

    /// <summary>
    /// This method is called for every Lambda invocation. This method takes in an SNS event object and can be used 
    /// to respond to SNS messages.
    /// </summary>
    /// <param name="evnt">The event for the Lambda function handler to process.</param>
    /// <param name="context">The ILambdaContext that provides methods for logging and describing the Lambda environment.</param>
    /// <returns></returns>
    public async Task FunctionHandler(SNSEvent evnt, ILambdaContext context)
    {
        foreach(var record in evnt.Records)
        {
            await ProcessRecordAsync(record, context);
        }
    }

    private async Task ProcessRecordAsync(SNSEvent.SNSRecord record, ILambdaContext context)
    {
        var feature = new Document
        {
            ["Id"] = Guid.NewGuid(),
            ["Description"] = record.Sns.Message
        };

        var table = Table.LoadTable(dynamoDBClient, tableName);

        await table.PutItemAsync(feature);

        context.Logger.LogInformation($"Processed record {record.Sns.Message}");

        await Task.CompletedTask;
    }
}

A message pushed in the topic will generate an SNS event, which, in turn, will trigger my lambda function. The lambda function will then store an item in the DynamoDB table. The item ID is a generated GUID (Globally Unique Identifier), and the description is the SNS message.

After deploying your functions, you should have a list like mine on the AWS console. I have called my functions Customer1, Customer2, and Customer3. In a real-world application, you should give a meaningful name that depicts your function purpose.

One last task here: copy your lambdas function ARNs in a notepad. You will use it in the coming step.

⚠️ Important Note:

You might permit your lambdas to interact with DynamoDB. Otherwise, they cannot store the items in the tables. Here are the steps to achieve that:

1. Create a role in IAM (Identity and Access Management)

2. Add an AmazonDynamoDBFullAccess policy to the role permissions, besides the default lambda execution permission

3. Edit your lambdas and Assign the created role to them

4. Configure the subscriptions

The most important task is now: subscribe your lambda functions to the SNS Topic.

Go to the SNS page in the AWS console and click on the FeatureDispatcher topic you created earlier. At the bottom, navigate to the Subscriptions tab and click "Create subscription"

In the Protocol dropdown list, select AWS Lambda.

In the Endpoint input, paste the ARN of your Customer1 lambda function.

The following section is where you define the filter policy for the subscription. First, click to enable the filter policy.

Choose the Message attributes option for the policy scope

In the JSON editor, type this:

{
  "feature_owner":["global", "customer1"]
}

With that policy in place, the Customer1 lambda function will only process messages within the topic if they have the feature_owner attribute set to"global" or "customer1". The lambda will ignore any message lacking this attribute or containing other values.

Leave the last section as it is and click Create Subscription. A confirmation message is displayed in the console if everything goes well.

As you might guess, you will repeat the process to create Customer2 and Customer3 lambdas subscriptions. Do not forget to paste the right ARN et to update the policy accordingly. For Customer2, you have:

{
  "feature_owner":["global", "customer2"]
}

And for Customer3:

{
  "feature_owner":["global", "customer3"]
}

Congratulations, you created all you need! Now, let's test if everything is working fine.

5. Let's test!

On the FeatureDispatcher topic page, click the Publish Message button at the top right. Enter a message body and a message attribute, as below. Here, You are simulating a global feature deployment, which should be stored in all the DynamoDB tables.

Click "Publish message".

Go to the DynamoDB page in the "Explore items" section. Refresh the items list and you should see an item in customer1 table. The description is the message you published some seconds ago from SNS.

Select the other tables. You should have the same item. because you have assigned the feature_owner attribute with the "global" value, each lambda function consumed the SNS message.

Now, let's issue another message publication. In this instance, ensure that the feature_owner attribute is set to "customer2". Publish the message.

Now, in DynamoDB items, there are two occurrences in the customer2 table. On the contrary, customer1 and customer3 still have only one occurrence each. That indicates the message was consumed by Customer2 lambda only, because of the attribute filter policy. The other lambdas ignore the message even if they subscribe to the topic. "customer2" is not part of their possible feature_owner attribute values.

Here is a side-by-side comparison between customer1 and customer2 tables:

Final Congratulations! The filter policies are working well. Lambda functions take them into account and process messages accordingly. The message filtering feature helps us build a more dynamic architecture using fewer resources. 👏👏

Going further ...

Of course, in this article, we simulate a foundational usage of the SNS message filtering feature, using the OR logic. In a real architecture, you can build more complex filter policies with more than one attribute and many more filter operators. Refer to the AWS documentation about subscription filter policies.

Conclusion

In this article, you've learned what the AWS SNS Message Filtering feature is and how to use it in a simulated multi-tenant architecture. By defining the appropriate filter policies in the subscriptions, you see that subscribers can consume only a subset of messages arriving at the topic. As a result, you will use fewer resources to address the sometimes complex requirements of your architecture.

I hope you like this article. Do not hesitate to leave feedback or a comment. If you have any questions about this article, feel free to ask. Thank you and happy coding!