Embedded Databases

When considering constructing a system, we often default to using complex databases like MySQL, PostgreSQL, Oracle DB, SQL Server, or MongoDB without pausing to consider if a simpler system would suffice for managing our data. These have become the default norms, however, there are numerous other databases that can address your issue with minimal impact on your application's complexity.

Embedded Databases

Embedded Databases are a type of database solutions which run in same process as your host application. Do not confuse it with databases utilized in embedded systems, as they are very different in both design and operation.
We are all familiar with SQLite database but have never really delved into its details. It is one of the databases that are integrated. It provides energy to many applications on your mobile devices, particularly Android devices.

How Embedded DB's Differ from the De-facto options

The majority of databases we encounter today utilize client-server architecture. Therefore, they need a separate process to operate their backend, handling the data and demanding additional resources. Typically, they are usually hosted on different servers in the real world, which can be quite expensive.

An embedded database is closely incorporated into an application software, meaning it is part of the application itself (rather than being a separate application) and operates within the same process as the host application, eliminating the need for inter-process communication and resulting in high throughput.

Advantages of Embedded DB’s

1. Performance: As they don’t require network calls to execute commands and generate result they are more performant

2. Easy Deployment: They can be bundled with the application and easily launched on end devices, unlike regular databases that require installation on the end devices which can cause compatibility issue if DB doesn’t support the end user device

3. Offline Access: Even if networks goes down we can access the data as it resides on same process as the application.

4. Data Security and Isolation: They are typically not directly accessible from outside the application, enhancing security by reducing the attack surface. This isolation is beneficial for handling sensitive data

5. Low Resource Consumption: A lot of embedded databases have minimal storage needs, typically under 1 MB, allowing your application to run on older or resource-limited devices.

Disadvantages of Embedded DB’s

1. Limited Scalability: Scalability is often constrained as embedded databases operate within a single application instance. They may struggle to handle large volumes of data or numerous concurrent users effectively

2. Lack of Advanced Features: Many embedded databases may lack features such as robust ACID transaction support, sharding, or advanced indexing capabilities that are common in traditional client-server databases.

3. Concurrency Limitations: While they can manage multiple threads, performance can degrade under heavy load due to simpler concurrency control mechanisms, which may lead to blocking during write operations

4. Data Recovery Challenges: In cases of crashes, ensuring data integrity can be more challenging compared to dedicated database systems that have built-in recovery mechanisms

5. Maintenance Overhead: Although they require less administration than client-server databases, any issues that arise must be handled within the application context, which can complicate troubleshooting and updates

Where they can be used

1. Read & Write Heavy System: They work well in applications with a high volume of reading, writing, or both. By running in the same process as the application, it reduces memory costs and latency incurred from network calls, ultimately improving throughput and latency.

2. Intermediatory Storage Engine: If your main database is overwhelmed, an embedded database can serve as a log. Your application writes data to an embedded database, while another process retrieves data from the database and writes it to the main database in the background.

3. Resource constraint Environments: They can be used in devices where their is minimal resource available as most of embedded DB’s are just library and have memory footprint of less than 5 MB

4. Offline First Systems: If data needs to be accessible when the device is offline, an embedded database can be used as it runs on the end device and does not rely on a network to function.

5. Backup or Replica Systems: As they have minimal footprint they can be used as backup system where high availability and data integrity are critical

6. Isolated Systems: It can be used if you don’t want another application to access the data of another application or your application don’t want to directly expose your data to outside world.

7. Cache System: It is suitable for cache implementation due to its high throughput and strong read and write speeds.

Popular Embedded DB

1. SQLite: The most popular embedded database, having relational model and can be queried using SQL.

2. Level DB: On disk Key Value store by Google built for performance

3. Rocks DB: A persistent key value store built for fast storage

4. Berkeley DB: A key value store which supports transaction, locking and replication

5. Unqlite: NoSQL and Key value store with support of transactions.

Conclusion

While traditional databases like MySQL, PostgreSQL, and MongoDB are widely used, embedded databases offer a simpler, faster, and more resource-efficient alternative for many use cases. However, they do come with limitations in scalability and concurrency handling. Understanding the trade-offs and matching the database to your specific needs is crucial in making the right choice for your application.