Quick POC: "Master-Follower Replication" using RocksDB

High Level Design

RocksDB is a high-performance embedded database for key-value data, designed for fast storage. In this post, I'll show you how to create a simple master-follower replication setup using RocksDB's backup and restore functionality.

POC

We'll create a simple system where:

• A master node writes data and creates periodic backups

• Follower nodes restore from these backups to replicate the data

• We'll verify the replication by writing data to the master and checking it on the followers

Install RocksDB

• brew install rocksdb

• let's create the necessary directories

mkdir -p /tmp/rocksdb_master /tmp/rocksdb_backup /tmp/rocksdb_follower1 /tmp/rocksdb_follower2

Create Master Script master.sh

#!/bin/bash

# Initialize Master DB
rocksdb_ldb --db=/tmp/rocksdb_master --create_if_missing put system:init "initialization"

# Write initial data
rocksdb_ldb --db=/tmp/rocksdb_master put user:1 Alice
rocksdb_ldb --db=/tmp/rocksdb_master put user:2 Bob

echo "Master: Initial data written."

# Periodically create backups
while true; do
  echo "Master: Creating backup at $(date)..."
  rocksdb_ldb --db=/tmp/rocksdb_master backup --backup_dir=/tmp/rocksdb_backup
  echo "Master: Backup completed"
  sleep 10
done

Create Follower Script follower.sh

#!/bin/bash

FOLLOWER_NAME=$1  # Pass "follower1" or "follower2" as an argument

# Initialize Follower DB
rocksdb_ldb --db=/tmp/${FOLLOWER_NAME} --create_if_missing put system:init "initialization"

# Periodically restore from backup
while true; do
  echo "${FOLLOWER_NAME}: Restoring from backup at $(date)..."
  rocksdb_ldb --db=/tmp/${FOLLOWER_NAME} restore --backup_dir=/tmp/rocksdb_backup
  echo "${FOLLOWER_NAME}: Restore completed"
  sleep 10
done

Terminal 1: Start the Master

./master.sh

You should see output indicating the initial data has been written and backups are being created periodically.

Terminal 2: Start Follower 1

./follower.sh follower1

Terminal 3: Start Follower 2

./follower.sh follower2

Both follower terminals should show periodic restore operations.

Testing the Replication

Terminal 4: Add Data to Master

# Add a new user
rocksdb_ldb --db=/tmp/rocksdb_master put user:3 Charlie

# Verify it was added
rocksdb_ldb --db=/tmp/rocksdb_master get user:3

You should see "Charlie" as the output from the get command.

Terminal 5: Verify Data on Followers

Wait at least 10-20 seconds for the replication to occur, then:

# Check on follower1
rocksdb_ldb --db=/tmp/follower1 get user:3

# Check on follower2
rocksdb_ldb --db=/tmp/follower2 get user:3

Both commands should output "Charlie", confirming that the data has been replicated to both followers.

How It Works

This simple replication system works through:

1. Periodic Backups: The master creates a backup of its database every 10 seconds

2. Periodic Restores: Each follower restores from the latest backup every 10 seconds

3. Eventual Consistency: Changes on the master will appear on followers within 10-20 seconds

Note

• This is an eventual consistency model with a potential lag of up to 20 seconds

• Followers are read-only - all writes must go to the master