Cap Theorom & Db Sharding

CAP THEOROM :

C – Consistency

A – Availability

P – Partitioning

The CAP Theorem is a fundamental concept in distributed system design, helping engineers understand the trade-offs between Consistency, Availability, and Partition Tolerance in any distributed data store.

Here’s a brief explanation of each:

1. Consistency (C): Every read receives the most recent write or an error. All nodes return the same data at the same time.

2. Availability (A): Every request (read/write) gets a response, even if it's not the most up-to-date data.

3. Partition Tolerance (P): The system continues to operate despite network partitions (i.e., communication between nodes may fail).

Key Insight of CAP Theorem:

• You can only guarantee two of the three properties (Consistency, Availability, Partition Tolerance) in a distributed system. This is often summarized as “Choose two out of three.”

For instance:

• CA Systems: Prioritize consistency and availability but struggle with partition tolerance. Example: Traditional relational databases.

• CP Systems: Focus on consistency and partition tolerance, sacrificing some availability during partition events. Example: HBase.

• AP Systems: Ensure availability and partition tolerance but may not always be consistent. Example: Cassandra, DynamoDB.

Real-World Application:

In practice, no system can completely sacrifice partition tolerance (since network failures are inevitable), so the choice typically boils down to balancing consistency and availability.

Understanding the CAP Theorem helps engineers make informed decisions about the right database or system architecture based on application requirements and trade-offs. It's particularly important when scaling systems across multiple regions or designing high-availability architectures.

Distributed System : A System consisting of a group of machines working in coordination so as to appear as a single coherent system to the end-user

Consistency : Any Read that is happening after a latest write , all the nodes should return the latest value of that Write.

Availability : Every Available node in the system should respond in a non error format to any Read request without the guarantee of returning the latest Write

Partition Tolerance : System will be responding to all Read and Write if the communication channel (or middleware) between nodes is broken ( or partitioned)

In CAP THEOROM generally will support any two of them in CAP , like C&A , A&C , A&P , P&C any one of the thing we need to be left , not all the time the Distributed will support C,A,P will accept anyone of the thing we have to left.

DB SHARDING :

Horizontal Partitioning Is same as Sharding used to store the large data with highly available and scaling

Local Sharding – Physical Machine (Physical DB)

Physical Sharding have Multiple Number of Local Sharding and the Data should be Stored in Multiple Physical Machines ( DB)

Advantage of Sharding :

Query Optimization

Better Performance

Reduce Latency

There are two types of Sharding

1.Algorithmic Sharding – App/Client knows which Shard the query has to go that’s Algorithmic Sharding

2.Dynamic Sharding – Other Module / Part which Client has to query in order to findout which Shard the query has to go to its called Dynamic Sharding

Disadvantage of Sharding :

Partition could be proper

Recover the old data , with Sharding to Non Sharding

Key Based Sharding :

Shared Key its should be an Static it won’t be change

We can select Multiple Column as Shared Key

First step here to pickup an Shard Key , Shard Key not equal to Primary Key

A Primary Key of the Table can be an Shard Key but the Vice Versa its Not Possible

Choosing the Shard Key means Pickup an Column on bases of which you are going to divide the data

Ex : UserId your picking up as Shard Key

After selecting the Shard(UserId) key It will use the Hash function so based on the Shard Key it will be give the Shard Value

Its an Algorithmic Sharding

Advantage :

Evenly Distributed Data

Disadvantage :

Adding new Shard because the Hashing function might Change

Range Based Sharding :

You have to store the Data in Shard in basis of Range

Its an Algorithmic Sharding

Ex : You are storing the value in Shard based on Month Range , like Each 2 Months Data should be Stored in One Shard

Advantage :

Same Database Schema for all Logical & Physical Shard

Disadvantage :

Increased Storage on Specific Shard This is called as an HOTSPOT

Data will not be uniformly distributed , like its an Range Based Shard so In Specific Range only it will be Shard to Specific Shard

Directory Based Sharding (Dynamic Sharding) :

Here Dividing your Shard Based on the Specific Column

Ex : Here you are Dividing Shard Based on Zone

Zone : 1 2 3 4 Shard : A B C D

Here Zone 1 —> Shard A , 2 —> B , 3 —> C , 4 —> D

These will be stores in Lookup Table based on that Zone the Shard could be Directed.

Advantage :

We can add Shard without Touching previous Shard

Remove Shard when ever you want

Data Evenly Distributed

Disadvantage :

For Every Read & Write the Application It will be first Connect with Lookup Table then Lookup Table Routs Where Read & Write has to go.

Lookup Table Crashes the Whole System will Crashes