Log-Structured Merge (LSM) Compaction Strategies: Leveled vs. Tiered

Core concept
An LSM tree (a write-optimized storage structure that batches writes in memory then flushes sorted files to disk) doesn't just accumulate files forever — it periodically merges and rewrites them in a process called compaction. Without compaction, you'd have hundreds of small sorted files (called SSTables — immutable sorted files on disk), and every read would scan all of them. The two dominant strategies for how to compact — leveled and tiered (size-tiered) — make fundamentally opposite bets about whether your workload is read-heavy or write-heavy, and picking the wrong one can quietly destroy your performance at scale.

flowchart TD
    A[MemTable: in-memory buffer] -->|flush when full| B[L0: small SSTables]
    B -->|compact into| C[L1: fixed-size level]
    C -->|compact into| D[L2: 10x larger level]
    D -->|compact into| E[L3: 10x larger level]
    B -->|tiered alternative: merge same-size files| F[Larger SSTable tier]

Concrete real-world example
RocksDB (an embeddable key-value store used inside MySQL, Cassandra, and CockroachDB) defaults to leveled compaction. Each level has a size cap — L1 might hold 256 MB, L2 holds 2.56 GB, L3 holds 25.6 GB, each 10× the previous. When L1 overflows, one SSTable from L1 is merged with every overlapping SSTable in L2, and the result is written back to L2. The key property: at any level ≥ L1, key ranges don't overlap across SSTables, so a read only needs to check at most one SSTable per level — usually 5–7 files total regardless of database size.

Tiered compaction (used by default in Apache Cassandra — a distributed wide-column database) takes the opposite approach: files aren't merged into a fixed-level structure. Instead, SSTables of similar size are grouped into tiers, and when a tier accumulates enough files (say, 4), they're merged into one bigger file that joins the next tier. No cross-file merging happens until that threshold, so writes are far cheaper — but reads may now scan all files in a tier because their key ranges do overlap.

One trade-off / gotcha
Leveled compaction dramatically reduces read amplification (number of files checked per read) — typically 10× better than tiered — but it multiplies write amplification (bytes written to disk per byte inserted), often by 10–30×. That means on a write-heavy workload, leveled compaction can saturate your disk I/O budget just on compaction, starving real writes. Tiered compaction writes less but can spike read latency unpredictably when a key happens to live in many overlapping files across a large tier.

An interview-style question to ponder
You're designing a time-series metrics database that ingests 500,000 data points per second and mostly serves dashboard queries that aggregate the last 5 minutes of data. Old data is queried rarely. Which compaction strategy would you choose, and what configuration detail would you adjust to make it practical?