How (and why) we brought SQLite to the Cloud

Simplicity, efficiency, and reliability. For the last 24 years, these have been the guiding principles of the world’s most popular database engine, SQLite. SQLite was originally devised as a way for clients to talk directly to disk without the need for a standalone database server. This lightweight design makes it the go-to database engine for desktop, mobile, and IoT application development. And for 24 years, that’s what SQLite was - an exceptional client-side library for relational data management.

At SQLite Cloud, we’re re-imagining what SQLite can be. Joined by Dr. Richard Hipp, creator of SQLite, we are building a distributed, scalable, high concurrency cloud database on top of 100% open source SQLite. Today, we’re excited to announce SQLite Cloud is entering open beta, and we’d like to take you behind the scenes on what we’ve been building.

Bringing SQLite to the Cloud

When we set out to build SQLite Cloud, it was important to us that we stay true to the original design philosophy of SQLite. Our system design decisions all center around simplicity and efficiency. To that end, we chose to base our distributed system on the Raft consensus algorithm.

The Raft Consensus Algorithm

The Raft consensus algorithm is a simple and understandable consensus mechanism for managing a replicated log across nodes. It brings fault-tolerance and strong consistency to SQLite Cloud with minimal overhead. To implement the Raft protocol, we chose the etcd/raft library. We favored this package because of its minimalistic design. It only implements the core raft algorithm - leaving network and disk I/O concerns to the user. This allowed our team to supply our own high-efficiency transportation and storage layer on top of the library.

The Transportation and Storage Layer

In the cloud, you’re only as fast as your network. So, we implemented a highly optimized network layer in C, capable of handling thousands of concurrent socket connections. Its efficiency stems from unique OS TCP optimizations, like kqueue on BSD systems, epool on Linux, and I/O completion ports on Windows. As for the storage layer - we couldn’t do better than the unmodified SQLite source code. In SQLite Cloud, you’re talking directly to SQLite over the network - and it’s reflected in SQLite Cloud’s performance.

Multithreading and Concurrency

From the beginning, our core process was engineered to be non-blocking and multi-threaded. Historically, SQLite was somewhat limited in write performance, as your database file is locked during a write. In SQLite Cloud, the system only waits the time required to write a commit message before handling another write. This mechanism is significantly faster than writing to the database file directly, doing away with the old adage that SQLite can’t support write-heavy workloads.

The final result of our implementation is a distributed and persistent cloud database, with in-memory performance. Think the speed of Redis, but with full Postgres-style SQL support. To see an example of this, you can check out our documentation site, where we used the FTS5 SQLite extension and SQLite Cloud to get RediSearch-like performance for our full-text search.

It’s not just fast - it’s also efficient. In our initial tests, SQLite Cloud required less than 10% of the memory Postgres required, given the same workload. For example - a SQLite cluster with 200 connections required ~100Mb of memory. As a point of reference, the default amount of memory allocated to a single Postgres instance’s shared buffers is 128Mb.

SQLite Cloud never forks a new process, but uses thread and connection pools out-of-the-box. We believe this efficient architecture will lead to significant cloud savings and performance improvements, and we’ll be publishing our benchmark findings in the coming weeks (you can subscribe to our newsletter here).

New Use Cases, Same SQLite

Pub/Sub

Many modern applications demand real-time updates, and this is particularly true for mobile applications. That’s why we introduced the Pub/Sub feature to SQLite Cloud. Pub/Sub lets applications subscribe to real-time updates whenever data changes in their SQLite instance. It’s scalable, allowing multiple subscribers to receive updates without impacting performance. It even allows subscribers to customize the types of messages they receive, which you can use to reduce network traffic and improve application performance.

Offline Synchronization

When your application has a network connection, synchronizing state is relatively easy. Things get much more complicated offline - many mobile apps will simply stop working properly without a stable connection. For future mobile development, we believe SQLite Cloud’s offline synchronization capabilities will be a game-changer. Offline synchronization enables client-side applications to continue operating as usual without a connection, then seamlessly synchronize once a network connection is reestablished. This is accomplished through the use of CRDT technology, which guarantees eventual consistency in a distributed system. In practice, this means using SQLite Cloud to build something like a shareable, online/offline text editor would require augmenting just one column in your database.

Weblite

Historically, SQLite has not been able to meet the demands of traditional web-based OLTP workloads. Using SQLite on a website typically involves adding some server-side infrastructure or sending large WASM packages to the client. To address these limitations, we’ve developed Weblite. With Weblite, getting a globally distributed SQLite database for your web app is as simple as inserting an html tag.

The Future is Lite

At SQLite Cloud, we believe SQLite is the simplest, most efficient and reliable platform on which to build the next generation of client-centric applications. We also believe that SQLite Cloud’s performance characteristics and efficiency make it a compelling choice for traditional server-centric OLTP workloads. In an era of consolidation and unification, we think SQLite Cloud provides a powerful way for teams to reduce their database sprawl.

Together, let's unlock a new era of magical, reactive, and lightning-fast applications, everywhere.

To try out what we’ve been working on yourself, join the beta for free at https://sqlitecloud.io/.

Until next time!