Implementing Cursor-Based Pagination with APPSYNC_JS and DynamoDB

TL;DR

This article offers insights on implementing cursor-based pagination with DynamoDB and APPSYNC_JS by setting up a serverless application, creating a basic schema, building resolvers, and deploying the service using Serverless Framework and AppSync Plugin. Test the pagination by seeding data and querying records with nextToken.

Introduction

With great Serverless power, comes great integration responsibilities

- Cloud Hashira 🥋

In this article, we'll explore the powerful capabilities of Amazon DynamoDB and its seamless integration with a wide range of applications and use cases. While its flexibility is impressive, certain compromises are necessary as some operations that are simple with databases like DocumentDB or RDS require additional steps and methods.

We'll guide you through implementing Cursor-Based pagination with DynamoDB and APPSYNC_JS runtime, providing valuable insights along the way.

APPSYNC_JS? That a thing?

On Nov. 17, 2022, AWS published the release of Javascript runtime, enabling developers using APPSYNC to easily test, evaluate, and debug their code in their local environment, and also provide more flexibility to define data access business logic while creating their resolvers. New to APPSYNC_JS? You can learn more here.

Pagination with DynamoDB

By default, AWS DynamoDB supports cursor-based pagination. This means that when you run a query in DynamoDB with the potential to return a large number of records, by passing a limit in your request without passing a value for ExclusiveStartKey or nextToken (in the case of AppSync) it fetches the records that match the condition and keys in your query from the first match to that within the specified limit. Along with the records, a LastEvaluatedKey or a nextToken is returned if there are still records that match your query but are outside the specified limit.

This key or token simply acts as a pointer or a cursor on where the next query operation is going to start fetching records. For flexibility, scalability, and efficiency, DynamoDB implements this type of pagination perfectly with some trade-offs. Due to the way it is implemented, you cannot:

• Skip to a specific page

• View the total number of pages

• Generally, sort features are limited

With that being said, let's build our solution.

Building the Solution

Basic Setup

We will be building our solution using Serverless Framework, Nodejs, and of course, an AWS account. I will assume that you have aws cli installed and your credentials configured. If not, check out how to install and set up aws cli to be on the same note.

Folder Set Up

From your terminal, run the following commands:

mkdir appsync_pagination_ddb
cd appsync_pagination_ddb
npm init -y
mkdir src
mkdir schema
touch serverless.yml
touch .eslintrc.json

Installing Required Packages

npm i --save-dev serverless-appsync-plugin
npm i --save-dev @aws-appsync/eslint-plugin @aws-appsync/utils
code .

In your .eslintrc.json paste the code below. The appsync eslint-plugin serves as a syntax validator for APPSYNC_JS runtime.

{
  "extends": ["plugin:@aws-appsync/base"]
}

Defining Our Schema

We will create a basic Book type for our schema. This will include a single Query resolver for listing records. Follow the steps below to create the necessary files and their corresponding contents.

schema/types.graphql

type Book {
  id: ID!
  title: String!
  author: String!
  copies: Int
  createdAt: AWSDateTime
  updatedAt: AWSDateTime
}

type ListBooksResult {
  books: [Book!]!
  nextToken: String
}

type Query {
  listBooks(input: ListBooksInput): ListBooksResult!
}

schema/inputs.graphql

input ListBooksInput {
  nextToken: String
  limit: Int
}

Creating the Resolvers

As defined already in schema/types.graphql, we will be creating one query resolver to list and paginate items from the database. The resolver would simply map out inputs from the appsync API to DynamoDB params in the request function and the corresponding result in the response function. Follow the steps below to create the necessary files and their corresponding contents.

src/resolvers/listBooks.js

import { util } from "@aws-appsync/utils";

export function request(ctx) {
  return {
    operation: "Scan",
    limit: ctx.args.input.limit ?? 10,
    nextToken: ctx.args.input.nextToken
  }
}

export function response(ctx) {
  if (ctx.error) return util.error("Books Not Found");
  return {
    books: ctx.result.items,
    nextToken: ctx.result.nextToken
  };
}

As previously discussed, cursor-based pagination is implemented by default in DynamoDB. In the listBooks.js resolver, the request function accepts the limit and nextToken parameters. nextToken is always null for the initial request or if you want to start fetching from the beginning of the records. the response function returns the records within the specified limit that matches the query. If there are still records that match the query, a signed value is returned for nextToken which the DB will use to return the next set of records.

IaC with Serverless Framework and AppSync Plugin

NB: The serverless-appsync-plugin version that supports APPSYNC_JS runtime is currently v2xx. It has slightly different configuration rules in comparison with v1xx. Check out the differences and guidelines on how to upgrade from v1xx to v2xx here.

Align your serverless.yml with the code below for infrastructure setup.

service: appsync-pagination-dynamodb
frameworkVersion: "3"
configValidationMode: error

plugins:
  - serverless-appsync-plugin

provider:
  name: aws
  region: us-west-2

appSync:
  name: pagination-dynamodb-api
  schema:
    - schema/*.graphql
  logging:
    level: ERROR
  authentication:
    type: API_KEY
  apiKeys:
    - name: myApiKey

  dataSources:
    library:
      type: AMAZON_DYNAMODB
      config:
        tableName: !Ref Library

  resolvers:
    Query.listBooks:
      functions:
        - dataSource: library
          code: src/resolvers/listBooks.js

resources:
  Resources:
    Library:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: library-${sls:stage}
        BillingMode: PAY_PER_REQUEST
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH

Deploy Service

Run the command below to deploy the service to your AWS account.

sls deploy -s dev --verbose

After successfully deploying the service, turn on your GraphBolt and take the resolvers for a spin. GraphBolt is your perfect AppSync companion. It holds all your APIs in one place and you can easily select, run, inspect, and debug your API operations.

You can use the AWS AppSync Console to test your API as well.

Testing

Seeding Data

Let us add some records to the DynamoDb we just created. Run the command below to fetch the data we are going to be writing to our database.

curl https://raw.githubusercontent.com/NwekeChidi/appsync_pagination/main/booksData.json --output booksData.json

Run the command below to create records in the DynamoDB table using the data in booksData.json file.

aws dynamodb batch-write-item --request-items file://booksData.json --output json

Testing the Resolver

Launch your GraphBolt and select the profile where the solution is deployed to if you have multiple profile credentials configured on your local, then select the corresponding API we just deployed, click the auth icon, select API Key and click save.

You can run the query listBooks when you are through creating the records to see the cursor-based pagination in play.

Passing the value of nextToken in the input for your next query returns the next set of records within the specified limit.

Conclusion

Implementing Cursor-Based Pagination with DynamoDB and APPSYNC_JS is a powerful and flexible technique for managing large datasets in serverless applications.

By following the steps outlined in this article, you can effectively set up your serverless application, create a basic schema, build resolvers, and deploy the service using Serverless Framework and AppSync Plugin. Keep in mind the trade-offs and limitations of this approach, and consider extending the practice with more advanced operations DynamoDB indexes for further optimization and faster data access.

To mitigate some of the cons presented by this operation, the client side of your application can persist the nextToken as page numbers and increment the total number of records per new fetch with the last returned token until the value for nextToken is null. This will enable users of your solution to skip between previously accessed pages.

You can find the complete code from this article on my Github.

Don't forget to clean up afterward. sls remove to the rescue!

Happy Coding and Sayonara!🙂