Integrating Amazon DocumentDB with Java Spring Boot and Amazon Lambda

DocumentDB

Amazon DocumentDB (with MongoDB compatibility) is a fully managed native JSON document database that makes it easy and cost effective to operate critical document workloads at virtually any scale without managing infrastructure.

Elastic Clusters:

Auto-scaling: Elastic Clusters automatically scale storage capacity, compute capacity, and instance count based on your workload demand.
Managed Infrastructure: Amazon DocumentDB manages the underlying infrastructure, including hardware provisioning, software patching, setup, configuration, and backups.
Single Endpoint: You connect to an Elastic Cluster through a single cluster endpoint.
Usage-based Pricing: You pay for the actual resources consumed based on the storage and compute capacity provisioned.
Ideal for Dynamic Workloads: Well-suited for applications with unpredictable workloads that require automatic scaling to handle varying demands.

Instance-based Clusters:

Manual Scaling: You need to manually provision and scale storage capacity and instance count as needed.
Managed Infrastructure: Like Elastic Clusters, DocumentDB manages the underlying infrastructure.
Multiple Endpoints: Instance-based Clusters provide separate reader and writer endpoints.
Fixed Pricing: You pay for provisioned resources regardless of actual usage.
Control Over Resources: Gives you more control over resource allocation but requires manual management.

Clusters

Volume

The cluster's data is stored in the cluster volume with copies in three different Availability Zones.
Amazon DocumentDB 5.0 instance-based clusters support two storage configurations for a database cluster: Amazon DocumentDB standard and Amazon DocumentDB I/O-optimized. For more information see

Notes **

Amazon DocumentDB (with MongoDB compatibility) clusters are deployed within an Amazon Virtual Private Cloud (Amazon VPC). They can be accessed directly by Amazon EC2 instances or other AWS services that are deployed in the same Amazon VPC.

Integration

Outside VPC

Since Amazon DocumentDB is VPC bound we need to create a SSH tunnel and port forwarding to connect to the DB from outside VPC

EC2 security group inbound rules should allow access SSH

Amazon DocumentDB SG inbound rule should allow access to 27107

ssh -i "ec2Access.pem" -L 27017:sample-cluster.node.us-east-1.docdb.amazonaws.com:27017 ubuntu@ec2-34-229-221-164.compute-1.amazonaws.com -N

Above Creates a SSH tunnel

mongosh --sslAllowInvalidHostnames --ssl --sslCAFile global-bundle.pem --username <yourUsername> --password <yourPassword>

Verify by connecting to mongo shell. On successful connection.

Inside VPC

DocumentDB can be accessed directly by Amazon EC2 instances or any other AWS services as long as they are in same VPC

Inside VPC with Lambda

Lambda's Internal VPC:

Lambda functions do run within an internal AWS-managed VPC, but it's not directly visible or configurable by users.
Lambda automatically provisions function inside the VPC and manages this VPC for you.
By default, Lambda functions can access the internet, but they can't access resources inside your VPC.
So we have to move the lambda inside documentDB VPC to make communication between the function and DB.
Once lambda function is moved into the VPC it losses the ability to communicate to the outside world
To make lambda function able to communicate to public internet we need to have a NAT Gateway.

Integrating with Java Spring boot

Connecting Outside VPC

You need to create SSH tunnel if you are connecting outside VPC. I have mentioned command to create tunnel in this document.

In the application.yaml file, we need to make a slight adjustment to the configuration. Specifically, the "host" key should point to localhost, and the "port" key should be configured according to the port forwarding settings.

Connecting within VPC

In the application.yaml file, we need to make a slight adjustment to the configuration. Specifically, the "host" key should point to documentDB host Since it's in same VPC not need of tunneling

Note:

DocumentDB requires a valid certificate for establishing connections.

Download the certificate from the link below https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem

application.yaml

spring:
  data:
    document-db:
      user: yourUserName
      password: yourPassword
      connection-string-template: mongodb://%s:%s@%s:%s/%s?retryWrites=false&directConnection=true&serverSelectionTimeoutMS=2000&tlsAllowInvalidHostnames=true&tls=true
      host: yourDocumentDBHost
      port: 27017
      db-name: yourDatabaseName

DocumentDBConfiguration

package com.platform.assetmanagement.utils.config;

import com.mongodb.ConnectionString;
import com.mongodb.MongoClientSettings;
import com.mongodb.reactivestreams.client.MongoClient;
import com.mongodb.reactivestreams.client.MongoClients;
import jakarta.validation.constraints.NotNull;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.SneakyThrows;
import lombok.extern.slf4j.Slf4j;
import org.apache.http.annotation.Contract;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.data.mongodb.config.AbstractReactiveMongoConfiguration;
import org.springframework.data.mongodb.core.ReactiveMongoTemplate;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManagerFactory;
import java.io.*;
import java.nio.file.Files;
import java.security.KeyStore;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.Stream;

@Slf4j
@Configuration
@RequiredArgsConstructor
public class DocumentDBConfig extends AbstractReactiveMongoConfiguration {

    private static final String CERT_FILE_PATH = "db-certs/global-bundle.pem";
    private static final String END_OF_CERTIFICATE_DELIMITER = "-----END CERTIFICATE-----";
    private static final String CERTIFICATE_TYPE = "X.509";
    private static final String TLS_PROTOCOL = "TLS";

    @Value("${spring.data.document-db.connection-string-template}")
    private String connectionStringTemplate;

    @Value("${spring.data.document-db.port}")
    private String port;

    @Value("${spring.data.document-db.db-name}")
    private String dbName;

    @Value("${spring.data.document-db.host}")
    private String host;

    @Value("${spring.data.document-db.user}")
    private String user;

    @Value("${spring.data.document-db.password}")
    private String password;

    @Override
    public MongoClient reactiveMongoClient() {
        try {
            return MongoClients.create(mongoClientSettings());
        } catch (Exception e) {
            // Print the error message
            System.err.println("Error occurred while creating MongoClient: " + e.getMessage());
            // You can also log the error using a logging framework like Logback or Log4j
            // logger.error("Error occurred while creating MongoClient", e);
            // Rethrow the exception or handle it according to your application's needs
            throw new RuntimeException("Error occurred while creating MongoClient", e);
        }
    }

    public MongoClientSettings mongoClientSettings() {
        return MongoClientSettings.builder()
                .applyConnectionString(new ConnectionString(getConnectionString()))
                .applyToSslSettings(builder -> {
                    builder.enabled(true);
                    builder.invalidHostNameAllowed(true);
                    builder.context(createSSLConfiguration());
                })
                .build();
    }

    @Bean
    public ReactiveMongoTemplate reactiveDocumentTemplate() {
        return new ReactiveMongoTemplate(reactiveMongoClient(), getDatabaseName());
    }

    @SneakyThrows
    private SSLContext createSSLConfiguration() {
        log.info("Reading AWS PEM certificate...");
//        ClassPathResource cpr = new ClassPathResource(CERT_FILE_PATH);
//        String certContent = Files.readString(cpr.getFile().toPath());

        InputStream inputStream = DocumentDBConfig.class.getClassLoader().getResourceAsStream(CERT_FILE_PATH);
        if (inputStream == null) {
            throw new FileNotFoundException("PEM file not found: " + CERT_FILE_PATH);
        }
        String certContent = new BufferedReader(new InputStreamReader(inputStream))
                .lines().collect(Collectors.joining("\n"));

        Set<String> allCertificates = Stream.of(certContent
                        .split(END_OF_CERTIFICATE_DELIMITER)).filter(line -> !line.isBlank())
                .map(line -> line + END_OF_CERTIFICATE_DELIMITER)
                .collect(Collectors.toUnmodifiableSet());

        CertificateFactory certificateFactory = CertificateFactory.getInstance(CERTIFICATE_TYPE);
        KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
        keyStore.load(null);

        int certNumber = 1;
        for (String cert : allCertificates) {
            Certificate caCert = certificateFactory.generateCertificate(new ByteArrayInputStream(cert.getBytes()));
            keyStore.setCertificateEntry(String.format("AWS-certificate-%s", certNumber++), caCert);
        }
        TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        trustManagerFactory.init(keyStore);
        SSLContext sslContext = SSLContext.getInstance(TLS_PROTOCOL);
        sslContext.init(null, trustManagerFactory.getTrustManagers(), null);
        return sslContext;
    }


    private String getConnectionString() {
        log.info("Generating connection string...");
        return String.format(this.connectionStringTemplate,
                this.user,
                this.password,
                this.host,
                this.port,
                this.getDatabaseName());
    }

    @Override
    @NonNull
    protected String getDatabaseName() {
        return this.dbName;
    }
}

2024-03-25T13:55:24.841+05:30  INFO 30508 --- [  restartedMain] c.p.a.utils.config.DocumentDBConfig      : Reading AWS PEM certificate...
2024-03-25T13:55:25.093+05:30  INFO 30508 --- [  restartedMain] org.mongodb.driver.client                : MongoClient with metadata {"driver": {"name": "mongo-java-driver|reactive-streams", "version": "4.11.1"}, "os": {"type": "Windows", "name": "Windows 11", "architecture": "amd64", "version": "10.0"}, "platform": "Java/Oracle Corporation/21.0.2+13-LTS-58"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=false, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='yourUserName', source='yourDB', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@12c05ea9, com.mongodb.Jep395RecordCodecProvider@bb2e83c, com.mongodb.KotlinCodecProvider@23f5c55c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='2000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=true, context=javax.net.ssl.SSLContext@19d683bd}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2024-03-25T13:55:25.523+05:30  INFO 30508 --- [  restartedMain] o.s.b.d.a.OptionalLiveReloadServer       : LiveReload server is running on port 35729
2024-03-25T13:55:25.898+05:30  INFO 30508 --- [localhost:27017] org.mongodb.driver.cluster               : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=541759900, setName='rs0', canonicalAddress=platformasset.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017, hosts=[platformasset.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017], passives=[], arbiters=[], primary='platformasset.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=null, topologyVersion=null, lastWriteDate=Mon Mar 25 13:54:54 IST 2024, lastUpdateTimeNanos=442201381626300}
2024-03-25T13:55:26.065+05:30  INFO 30508 --- [  restartedMain] c.p.a.utils.config.DocumentDBConfig      : Generating connection string...
2024-03-25T13:55:26.065+05:30  INFO 30508 --- [  restartedMain] c.p.a.utils.config.DocumentDBConfig      : Reading AWS PEM certificate...
2024-03-25T13:55:26.090+05:30  INFO 30508 --- [  restartedMain] org.mongodb.driver.client                : MongoClient with metadata {"driver": {"name": "mongo-java-driver|reactive-streams", "version": "4.11.1"}, "os": {"type": "Windows", "name": "Windows 11", "architecture": "amd64", "version": "10.0"}, "platform": "Java/Oracle Corporation/21.0.2+13-LTS-58"} created with settings MongoClientSettings{readPreference=primary, writeConcern=WriteConcern{w=null, wTimeout=null ms, journal=null}, retryWrites=false, retryReads=true, readConcern=ReadConcern{level=null}, credential=MongoCredential{mechanism=null, userName='yourUserName', source='yourDB', password=<hidden>, mechanismProperties=<hidden>}, transportSettings=null, streamFactoryFactory=null, commandListeners=[], codecRegistry=ProvidersCodecRegistry{codecProviders=[ValueCodecProvider{}, BsonValueCodecProvider{}, DBRefCodecProvider{}, DBObjectCodecProvider{}, DocumentCodecProvider{}, CollectionCodecProvider{}, IterableCodecProvider{}, MapCodecProvider{}, GeoJsonCodecProvider{}, GridFSFileCodecProvider{}, Jsr310CodecProvider{}, JsonObjectCodecProvider{}, BsonCodecProvider{}, EnumCodecProvider{}, com.mongodb.client.model.mql.ExpressionCodecProvider@12c05ea9, com.mongodb.Jep395RecordCodecProvider@bb2e83c, com.mongodb.KotlinCodecProvider@23f5c55c]}, loggerSettings=LoggerSettings{maxDocumentLength=1000}, clusterSettings={hosts=[localhost:27017], srvServiceName=mongodb, mode=SINGLE, requiredClusterType=UNKNOWN, requiredReplicaSetName='null', serverSelector='null', clusterListeners='[]', serverSelectionTimeout='2000 ms', localThreshold='15 ms'}, socketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=0, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, heartbeatSocketSettings=SocketSettings{connectTimeoutMS=10000, readTimeoutMS=10000, receiveBufferSize=0, proxySettings=ProxySettings{host=null, port=null, username=null, password=null}}, connectionPoolSettings=ConnectionPoolSettings{maxSize=100, minSize=0, maxWaitTimeMS=120000, maxConnectionLifeTimeMS=0, maxConnectionIdleTimeMS=0, maintenanceInitialDelayMS=0, maintenanceFrequencyMS=60000, connectionPoolListeners=[], maxConnecting=2}, serverSettings=ServerSettings{heartbeatFrequencyMS=10000, minHeartbeatFrequencyMS=500, serverListeners='[]', serverMonitorListeners='[]'}, sslSettings=SslSettings{enabled=true, invalidHostNameAllowed=true, context=javax.net.ssl.SSLContext@7fdd661e}, applicationName='null', compressorList=[], uuidRepresentation=UNSPECIFIED, serverApi=null, autoEncryptionSettings=null, dnsClient=null, inetAddressResolver=null, contextProvider=null}
2024-03-25T13:55:26.634+05:30  INFO 30508 --- [localhost:27017] org.mongodb.driver.cluster               : Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=REPLICA_SET_PRIMARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=13, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=406052800, setName='rs0', canonicalAddress=DocumentDB.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017, hosts=[Document.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017], passives=[], arbiters=[], primary='Document.ccouih7cdiv2.ap-northeast-1.docdb.amazonaws.com:27017', tagSet=TagSet{[]}, electionId=7fffffff0000000000000001, setVersion=null, topologyVersion=null, lastWriteDate=Mon Mar 25 13:54:55 IST 2024, lastUpdateTimeNanos=442202124708600}

Now that the connection has established we can do simple crud operation.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

UserEntity

import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;

@Document(collection = "users")
public class User {
    @Id
    private String id;
    private String name;
    private int age;
    // Getters and setters
}

UserRepository

import org.springframework.data.mongodb.repository.MongoRepository;

public interface UserRepository extends MongoRepository<User, String> {
}

UserService

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.List;

@Service
public class UserService {
    @Autowired
    private UserRepository userRepository;

    public List<User> getAllUsers() {
        return userRepository.findAll();
    }

    public User getUserById(String id) {
        return userRepository.findById(id).orElse(null);
    }

    public User createUser(User user) {
        return userRepository.save(user);
    }

    public User updateUser(String id, User newUser) {
        newUser.setId(id);
        return userRepository.save(newUser);
    }

    public void deleteUser(String id) {
        userRepository.deleteById(id);
    }
}

UserController

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;
import java.util.List;

@RestController
@RequestMapping("/api/users")
public class UserController {
    @Autowired
    private UserService userService;

    @GetMapping
    public List<User> getAllUsers() {
        return userService.getAllUsers();
    }

    @GetMapping("/{id}")
    public User getUserById(@PathVariable String id) {
        return userService.getUserById(id);
    }

    @PostMapping
    public User createUser(@RequestBody User user) {
        return userService.createUser(user);
    }

    @PutMapping("/{id}")
    public User updateUser(@PathVariable String id, @RequestBody User user) {
        return userService.updateUser(id, user);
    }

    @DeleteMapping("/{id}")
    public void deleteUser(@PathVariable String id) {
        userService.deleteUser(id);
    }
}

As you the see API is successfully creating document in the Document DB cluster.

Conclusion

In summary, integrating Amazon DocumentDB with Java Spring Boot and Amazon Lambda provides developers with a powerful and streamlined solution for building cloud-native applications. This combination offers compatibility, performance, simplified development, scalability, and cost-effectiveness, enabling developers to create resilient, scalable, and cost-efficient applications that meet modern business requirements.

Integrating Amazon DocumentDB with Java Spring boot and Amazon lambda

Table of contents