System Design (Day - 6)

NoSql Database Optimised (Writes), Data Consistency.

1. Let’s say We have some Insertion, selection sort of 4 queries to the databases when each time that query executes and changes in the database, the database will acknowledge with success or failure, we have 4 operations like that, so we can Condense that data query in to one query, so that instead of 4 read and write operations, we can do it in one Query, but it requires some additional memory in the server, with this there will be lesser Input and output operations to the Database.

2. We can use Log Using Linked list, with this one the write operations becomes O(1)Faster writes like if you know the Linked list data structures then you can imagine that the write operation becomes O(1),
   but read operations will become slow, because in linked list if you want to search something means it’ll take O(n) its linear, we can’t use like this for the read operations, instead of storing the linked list we can use the sorted array to store the data in smaller chunks, and then in the database we can do 2 or 3 smaller chunks and merge them and create a larger chunk so that it’ll be easy to do the read operations O(logn), with the help of chunks we can reduce the latency of read operation.

3. We are doing like merging the smaller chunks into bigger chunks, like 6 + 6 size of sorted array merged to sorted o 12 size one big chunk, and also we can attach the bloom filters to the each chunk,
   we can have false positive in the bloom filters, so if we reduce the false positive in the each chunk then we can reduce the unnecessary read operations on the chunks. that big chunks we can do binary search and get the fast read operations, Smaller chunks are called as Sorted string table, after merging the smaller chunks we get the bigger chunks that is called Compaction.

Data Consistency

What is Data consistency: If we have multiple copies of data in multiple database that should be in sync with each other, or the content should match each other, this is the general idea about data consistency.

1.If we have only one server and one Database that means there is high chance of single point of failure, which is really not good, and also costs high for vertical scaling and we have to stop at some point, like we don’t get much hardware support, at some point. Latency is also another problem,
For this Single point of failure problem we can have database servers that is deployed in different different regions, to solve the single point of failure. and Latency problem is also solved by this, because we have deployed our server in different regions.
2. To stay in consistent with each servers that is deployed in different regions, let’s say database A get and update, and it tells the database B to update, but the communication between the both database are not working, so database A should receive an Acknowledgement from the Database B so that the data has been updated successfully, but it never happens, for that the Database A has to sent the message recursively, but Database B never returns anything in the Acknowledgement to the Database A, this is a Two generals Problem, which is explained here very nicely here https://geeks.co.uk/2020/03/two-generals-problem/
3. 2 Phase commit: Whenever any write operation comes into the Leader Database, its will send the prepare request to he followers, and they’ll return with the acknowledgement, that is the 1st phase, and after receiving the acknowledgement, then the Leader will send commit message to its followers, if any of the Database fails to acknowledge that 1st phase, then every database has the rollback in their memory so that they can rollback into the previous state, including the Leader, if everything goes well then every follower databases will commit the updated data sent by the leader, so that every database will stay in consistent with each other, if commit message fails the the leader will retry until gets the success, or it’ll attempt for some time and give up and it’ll do rollback.

With all these things, the cost of the physical servers and databases increase, but the consistency increases, so that it’s very good for the database that has to be consistent at all the time. like Transaction etc…