← all lessons
System Design · Consistency & Replication ·

Read-Scale Fanout vs. Write Quorum: Why You Can't Just Add Replicas to Fix Both

The core concept explains a fundamental tension: the strategies that make reads fast and scalable are often the exact strategies that make writes slower and harder to keep consistent — and vice versa.

- Core concept

When you add more replicas to a cluster, reads get cheaper because more nodes can serve them in parallel. But each write now has to reach more nodes before it can be confirmed safe. If you require a strict quorum (majority of nodes must confirm a write before it's acknowledged) to prevent stale reads, every additional replica you add increases write latency and the blast radius of a slow node. The cruel irony is: naively scaling reads by adding replicas can degrade write throughput until the system falls over on the write side. This tension — read fanout versus write quorum — is one of the most common hidden failure modes in interview designs and real production clusters.

- Diagram

flowchart LR
    Client -->|write| Leader
    Leader -->|replicate| R1
    Leader -->|replicate| R2
    Leader -->|replicate| R3
    Leader -->|replicate| R4
    Leader -->|replicate| R5
    R1 -->|ack| Leader
    Leader -->|ack when quorum met| Client

- Concrete real-world example

Imagine a product recommendation service backed by a 5-node cluster running Cassandra (a distributed database using peer-to-peer replication). You start with a write quorum of 3 (W=3, meaning 3 of 5 nodes must confirm). Reads feel fresh and fast. Traffic grows, so an engineer adds 4 more nodes for read capacity, bringing the cluster to 9 nodes. Now W=3 out of 9 no longer guarantees you'll avoid stale reads — you need W=5 to maintain the same consistency guarantee. Every write now waits for 5 confirmations instead of 3. The 99th-percentile write latency jumps because a single slow node among the required 5 gates every write. The "scaling" operation made writes materially worse.

- One trade-off / gotcha

Tail latency (the slowest responses in a distribution, e.g. the 99th percentile) is the hidden enemy of large quorums. Because a quorum write can't complete until the slowest required node responds, adding replicas to the quorum set doesn't average out slowness — it maximizes it. With W=5, you're always waiting for the worst performer among five nodes. This is sometimes called the "weakest link" effect, and it means the gains from read scaling are often eaten by write latency degradation before you ever hit a storage bottleneck.

- An interview-style question to ponder

You're designing a global leaderboard for a multiplayer game. Writes happen constantly (millions of score updates per minute), and reads must be reasonably fresh — players can tolerate seeing a score that is at most 5 seconds stale. You have budget for 9 replica nodes spread across 3 geographic regions (3 nodes per region). How do you configure your read and write quorums, and how do you handle the cross-region replication lag without blocking every write on a global quorum?

Stuck? Show a hint

Think about whether the consistency requirement (5-second staleness budget) actually forces you to do a global quorum on every write, or whether you can exploit the staleness budget to decouple regional writes from cross-region replication.

Show answer

Use a local quorum (write confirmed by 2 of 3 nodes in one region, reads served from the same region) and let cross-region replication happen asynchronously within the staleness budget.

  • Writes go to the local region's 3 nodes, requiring W=2 (local quorum). This is fast — same-region round trips are ~1–5 ms. The write is confirmed to the client without waiting for the other two regions.
  • Async replication propagates the write to the other 6 nodes in the background. Cross-region latency is typically 50–150 ms, well inside the 5-second staleness budget players are promised.
  • Reads are served with R=2 from the local region (local quorum). Because writes were already confirmed locally with W=2, a local R=2 read will always see the latest local write. Staleness only exists across regions, not within a region — and the async lag is bounded well under 5 seconds.
  • But why not just use a global quorum (W=5 of 9) to guarantee perfect freshness everywhere? Because cross-region write latency alone is 50–150 ms minimum, and you'd be gating every score update on the slowest of five geographically dispersed nodes. At millions of writes per minute, that quorum would become your bottleneck almost immediately, and a single region's network hiccup would stall writes globally.
  • Watch out: if a full region goes dark, players in that region will be rerouted to another region and might briefly read scores that are up to the replication lag behind — make sure your SLA explicitly covers this edge case as an acceptable degradation, not a consistency violation.