How to Achieve Persistent and Fast In-Memory Databases

In the last few days, I have been creating an in-memory database server similar to Redis using GOLANG, along with a client (CLI is in progress) that communicates using RESP, a protocol used by Redis. If you want to check it out, here's a link: quickdb.

So, I was trying to figure out what would happen if the server crashes. In that case, my entire database would reset on restart, and all the data would be lost. We wouldn't be able to recover the data that was previously set. How can we prevent this from happening? How can we fix this?

Hmm, we also need to keep it fast so it doesn't affect the speed of the in-memory database. To achieve this, we mainly have two options: snapshotting and AOF (Append Only File, used by Redis). There are other options too, like write-ahead logs, but they slow down the process because they first log the changes in the database and then process them.

We can use snapshotting and append to a file, which can run in the background using Golang's GOROUTINES to asynchronously handle the task. With snapshotting, we can take a snapshot or copy the current state of the database and save it to a file or another database after a certain period. In case of a server outage, upon restarting, we can replay all the snapshots, and the database will return to its previous state. However, there's a problem: if the server goes down before a snapshot is taken, we won't have the latest state of the server in the snapshot, which defeats the purpose.

The same goes for appending to a file. In this method, we log all operations on the server or database into a log file asynchronously. Before starting the database, we replay these log files to restore the previous state. However, some operations might not be logged yet because it's asynchronous. Also, appending to a file is slower on restart since you need to replay all the events, whereas with snapshotting, you can directly apply the last snapshot to the database, and you're ready to go.

Hmm so to tackle the issue of state being not up to date, I thought about using a graceful shutdown technique with context in Golang to fix this issue. Whenever there's a problem with the server and we need to shut down, we can take a snapshot or append all pending operations before shutting down. This ensures the state remains up-to-date.

Here is a simple example that I implemented in Golang for my in-memory database:

package aof

import (
    "bufio"
    "io"
    "os"
    "sync"
    "time"

    "github.com/yagyagoel1/quickdb/utils"
)

type Aof struct {
    file *os.File
    rd   *bufio.Reader
    mu   sync.Mutex
}

func NewAof(path string) (*Aof, error) {
    f, err := os.OpenFile(path, os.O_CREATE|os.O_RDWR, 0666)
    if err != nil {
        return nil, err
    }
    aof := &Aof{file: f,

        rd: bufio.NewReader(f),
    }
    go func() {
        for {
            aof.mu.Lock()
            aof.file.Sync()
            aof.mu.Unlock()
            time.Sleep(time.Second)
        }

    }()
    return aof, nil
}
func (aof *Aof) Close() error {
    aof.mu.Lock()
    defer aof.mu.Unlock()

    return aof.file.Close()
}

func (aof *Aof) Write(value utils.Value) error {
    aof.mu.Lock()
    defer aof.mu.Unlock()
    _, err := aof.file.Write(value.Marshal())
    if err != nil {
        return err
    }
    return nil
}

func (aof *Aof) Read(callback func(value utils.Value)) error {
    aof.mu.Lock()
    defer aof.mu.Unlock()
    resp := utils.NewResp(aof.file)
    for {
        value, err := resp.Read()
        if err == nil {
            callback(value)
        }
        if err == io.EOF {
            break
        }
        return err

    }

    return nil
}

Here, as you can see, I am using my built-in function to convert the command to the resp type using value.Marshal(). I am also using a goroutine to add commands that modify the database. Additionally, I use a mutex to prevent critical section problems by locking the database during read or write operations to avoid any overwrites.

fmt.Println("listening on port 6379")
    l, err := net.Listen("tcp", ":6379")
    if err != nil {
        fmt.Println(err)
        return
    }
    aof, err := aof.NewAof("database.aof")

    if err != nil {
        fmt.Println(err)
        return
    }
    defer aof.Close()

    aof.Read(func(value utils.Value) {
        command := strings.ToUpper(value.Array[0].Bulk)
        args := value.Array[1:]
        handler, ok := handler.Handlers[command]
        if !ok {
            fmt.Println("Invalid command: ", command)
            return
        }

        handler(args)
    })

Here as you can see on starting the server i am reading back from the log file to get the previous state back before starting the server.

In my example of AOF, I haven't covered graceful shutdown, but it's easy to add, and I suggest you include it as well.

So, I hope you learned a lot from this blog. Thank you so much for your time. Until then, see you next time!

Yagya Goel
Github
Linkedin