DynamoDB Streams for no-code notifications via SNS

I have been working on implementing Event Driven Architecture (EDA) in AWS for an e-commerce client. I am new to EDA and am learning various aspects of it. I discovered several new services in AWS that I had never used before. Amazon DynamoDB is one of them.

Amazon DynamoDB is a serverless, NoSQL, fully managed database with single-digit millisecond performance at any scale. It overcomes the scaling and operational complexities of relational databases. For ex. DynamoDB delivers consistent single-digit millisecond performance for a shopping cart use case, whether you've 10 or 100 million users. I will cover power of DynamoDB in separatepost.

In this article, I will focus on setting up no-code notifications using an important feature of DynamoDB, DynamoDB Streams.

A DynamoDB stream is an ordered flow of information about changes to items in a DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table. DynamoDB Streams writes stream records in near-real time so that you can build applications that consume these streams and take action based on the contents.

Below is the outline of flow which will be implementing. Any change in DynamoDB table will be captured by DynamoDB Streams. EventBridge pipe will have DynamoDB Streams as source, filter will be used to match the trigger criteria for email notification using Amazon SNS.

1. Create DynamoDB table: Provide table name, partition_key and sort_key (optional).

   Partition key is basically table's primary key. You can use a sort key as the second part of a table's primary key. The sort key allows you to sort or search among all items sharing the same partition key.

   

2. Enable DynamoDB Streams: It is very simple, select your dynamodb table -> Exports and Streams -> Turn on DynamoDB Streams.
   It will ask you to choose one option, we will go with New and Old Images.

   This option will capture both old and new changes to the data items.

3. Create Amazon SNS topic:

   1. ProductDetails is the SNS topic name, it could be anything as per your implementation.

      #aws sns create-topic --name ProductDetails

   2. Enter the following command to subscribe an email address to PoductDetails. (Replace region and accountID with your Amazon Region and account ID, and replace talvinder@example.com with a valid email address.)

      #aws sns subscribe \ --topic-arn arn:aws:sns:region:accountID:ProductDetails\ --protocol email \ --notification-endpoint talvinder@example.com

   3. Amazon will send you confirm subscription on your email, accept that.

      

4. Create EventBridge Pipes:

   1. Go to EventBridge, select pipes. Select source as DynamoDB Streams, select the stream you have created earlier in pervious step in DynamoDB.

      

   2. You have option to directly select the Target endpoint which is SNS topic. Lets do that first and then will come back to Filtering.

      

   3. Now, click on Filtering, its took me a lot of time to understand and configure this. There are 2 things to undestand here.

      Sample Event - is the format of how AWS logs the change event.

      Event Pattern - is the matching of pattern based on criteria provided. you can read about this in detail here.

   4. I am using prefix matching criteria here. Keys is used to point to the DynamoDB table attribute for which you want to match. Here base_store field is matched if there is any new entry comes in with prefix Dummy it will trigger the notification.

          {
           "dynamodb": {
             "Keys": {
               "base_store": {
                 "S": [{
                   "prefix": "Dummy"
                 }]
               }
             }
           }
         }
      

   5. Save the filtering criteria.

5. Testing: Now we are done with no-code implementation, lets test now.

   Create an item in DynamoDB table.

   

   You should recieve a similar notification email. We can notice some important parameters like eventName:INSERT because we created a new item in the table. In case if we deleted it should have been eventName:REMOVED. If you remember we chose New and Old Image while enabling DynamoDB streams, therefore we have NEW_AND_OLD_IMAGES in the notification which will capture both values old and new.

   

6. Lets try to modify the DynamoDB item we created just now. Remember we will not modify the partitionKey. I am adding new attribute value as shown below.

   

   Now here is the SNS notification.

   

   I hope this helped you explore and implement DynamoDB Streams, which are more powerful than you might think.