System Design: Core Challenges in Building Web-Scale Systems

As I continue my journey through System Design School, one thing has become very clear: most system design decisions come down to handling scale.

The real challenges start showing up only when you're trying to serve millions of users reliably. Without that scale, you're mostly solving backend engineering problems—not true system design ones.

In this post, I want to walk you through the four core challenges that come up when designing web-scale distributed systems. These are the problems that reusable patterns like sharding, queues, replication, and consistency models are built to solve.

---

🧩 Challenge 1: Too Many Concurrent Users

🔥 The Problem

A single server can only handle limited requests per second (QPS). Once it hits that limit, performance degrades fast. Imagine millions of users trying to connect to your backend simultaneously.

🛠️ The Solution

We replicate our application logic and split the user load using load balancers. Similarly, for databases, we use read replicas to distribute the load.

🔁 Think of it like cloning your restaurant and seating guests randomly at each location. Everyone gets the same experience, just at a different table.

Tech Tools:

• Load Balancers (e.g., NGINX, HAProxy)
• Horizontal Pod Autoscalers (Kubernetes)
• DB Replication (Postgres, MySQL, MongoDB)

---

🧩 Challenge 2: Too Much Data to Fit on One Machine

🔥 The Problem

Once your app collects enough data (like tweets, logs, videos), it becomes impossible to store and process everything on one server.

🛠️ The Solution

We shard the data — partitioning it across different machines using a defined logic like user ID or region.

📦 Think of a library so big that you need separate buildings for each subject. Sharding is choosing which book goes where.

Tech Tools:

• Sharded NoSQL databases (Cassandra, DynamoDB)
• Consistent Hashing (used by systems like Redis Cluster, Kafka)
• Custom partitioning logic in relational DBs

---

🧩 Challenge 3: System Must Be Fast and Responsive

🔥 The Problem

Users expect your app to respond in < 500ms. While reads are usually fast (especially with caching), writes often take time due to the amount of coordination required.

🛠️ The Solution

We make the system asynchronous. When a user writes something, we acknowledge the request quickly and defer the actual processing to background jobs or message queues.

🕐 It’s like ordering coffee at Starbucks — you pay and walk away, while your drink is being prepared behind the counter.

Tech Tools:

• Message Queues (Kafka, RabbitMQ, AWS SQS)
• Background workers (Celery, Sidekiq)
• Async APIs with task queues

---

🧩 Challenge 4: Inconsistent or Outdated Data

🔥 The Problem

Replication and async writes lead to eventual inconsistency. You might see outdated data because changes haven't yet propagated everywhere.

🛠️ The Solution

We accept that eventual consistency is OK in many use cases. The system catches up over time. But for sensitive apps (like banking), we opt for strong consistency even if it’s slower.

⚖️ Would you rather see a 2-second delay or risk losing your bank transfer? Different apps, different trade-offs.

Tech Tools:

• Quorum reads/writes (e.g., in Cassandra)
• Vector clocks, CRDTs, versioning
• Strong consistency in ACID-compliant systems (e.g., traditional RDBMS)

---

✅ TL;DR: Building for Scale Means Trade-offs

To build systems that scale to millions of users and TBs of data, we:

1. Replicate logic and data for availability,
2. Shard data to distribute load,
3. Queue heavy writes to maintain responsiveness,
4. Tolerate eventual consistency where acceptable.

These trade-offs power platforms like Twitter, Slack, YouTube, and LinkedIn — and they're the foundations of modern backend architecture.

---

In upcoming posts, I’ll dive deeper into each solution:

• How load balancers work and what real traffic shaping looks like
• Real-world sharding strategies and failure stories
• When eventual consistency breaks user trust
• And how tools like Redis, Kafka, and async task queues help

If you're on a similar journey or want to discuss system design, follow me or drop a comment below 🚀