Token Bucket Rate Limiting: Shaping Traffic Before It Breaks Your Services

Core concept
When a service receives more requests than it can safely handle, you need a rule for which requests to accept and which to reject — not just a binary on/off switch, but a mechanism that tolerates short bursts while enforcing a sustained ceiling. A token bucket solves this: imagine a bucket that fills with tokens at a fixed rate (say, 100 per second) up to a maximum capacity (say, 200). Each incoming request consumes one token; if the bucket is empty, the request is rejected or queued. This gives you two independent levers — the refill rate controls your steady-state throughput, and the bucket capacity controls how large a burst you'll absorb before throttling kicks in.

flowchart LR
    A[Request arrives] --> B{Tokens available?}
    B -->|Yes| C[Remove 1 token]
    C --> D[Serve request]
    B -->|No| E[Reject / queue]
    F[Refill timer] -->|adds tokens up to max| B

Concrete real-world example
A payment API allows 50 transactions per second on average, but a merchant's checkout flow legitimately spikes to 150 TPS for 1–2 seconds at the top of an hour. With a token bucket of capacity 200 and refill rate 50/s, the bucket fills during the quiet period and can absorb that 2-second burst (200 tokens consumed) without dropping a single request. A simpler approach — a fixed window counter (resets every second) — would drop 100 of those 150 requests in that second, breaking real user checkouts.

One trade-off / gotcha
Token buckets are generous to bursts, which is usually what you want — but it means a client who deliberately waits to accumulate tokens can then hammer your service with a full-capacity burst all at once. If your downstream system (say, a database) cannot handle sudden spikes even briefly, you may need a leaky bucket (a variation that smooths output into a fixed-rate stream regardless of input) instead. The token bucket shapes average load; the leaky bucket shapes instantaneous load. Pick the wrong one and you've protected the wrong bottleneck.

An interview-style question to ponder
You're designing rate limiting for a public REST API (an interface where clients send requests over HTTP) shared by thousands of tenants (individual paying customers). You want per-tenant limits enforced consistently even when your rate-limiting service runs on 10 servers. Where do you store the token bucket state, and what consistency trade-off does that choice force you to make?