Fencing Tokens: Preventing Split-Brain Writes When a Lock Holder Comes Back Late

Core concept
When a distributed system grants a lock (exclusive permission to write) to a node, that node might pause — due to garbage collection, a network hiccup, or a slow disk — and by the time it resumes, the lock has already expired and been granted to someone else. Without a mechanism to detect this, two nodes can both believe they hold the lock and corrupt shared state simultaneously, a situation called split-brain. A fencing token solves this: every time a lock is granted, the lock service issues a monotonically increasing integer (a number that only ever goes up) alongside it. The storage system rejects any write whose token is lower than the highest token it has already seen, making stale lock holders harmless rather than dangerous.

flowchart LR
    A[Lock Service] -->|token=42| B[Node A gets lock]
    A -->|token=43| C[Node B gets lock]
    B -->|write + token=42| D[Storage]
    C -->|write + token=43| D
    D -->|reject token=42| B
    D -->|accept token=43| C

Concrete real-world example
Imagine a payment processor where exactly one worker node should write a "charge approved" record. Worker 1 acquires the lock and gets token 100, then pauses for 35 seconds during a Java garbage collection (automatic memory cleanup that can freeze a process). The lock expires after 30 seconds; Worker 2 acquires the lock and gets token 101. Worker 1 wakes up and tries to write with token 100. The database has already seen 101, so it silently drops Worker 1's write. The customer is charged exactly once.

One trade-off / gotcha
The storage layer itself must enforce the token check — the client cannot be trusted to self-police. If you're writing to a third-party service (like a cloud blob store) that has no concept of fencing tokens, this pattern cannot protect you. Leases plus idempotency keys (unique IDs per operation that deduplicate retries) are your fallback, but they guard against duplicates, not against a stale write that carries genuinely different data.

An interview-style question to ponder
A team argues: "We don't need fencing tokens — we use a 10-second lock TTL (time-to-live, the lock's expiration timeout) and our GC pauses never exceed 2 seconds, so the lock will always have expired before the old holder can act." Is this reasoning sound, and would you accept this system in production?