Spanner: Google's Globally-Distributed Database

The problem: a database that is global and still tells the truth

Google’s advertising business ran on a sharded MySQL deployment that took years to re-shard and could not survive a data centre outage without manual failover. The team wanted a database replicated synchronously across continents, with SQL and real transactions, that kept working when a whole region vanished. Bigtable offered the scale but no multi-row transactions; its successor Megastore offered transactions but wrote too slowly to be general. And one requirement seemed to rule out any global design: external consistency, which means that if transaction T2 starts after T1 has committed, anywhere on Earth, T2 must see T1’s writes and carry a later timestamp.

The obstacle is the one Lamport named in 1978. Timestamps from different machines cannot be compared, so a transaction committed in Tokyo and one committed in Dublin a millisecond later could easily be recorded in the wrong order, and a snapshot read in between would see an impossible history. Every previous system either routed all commits through one place, which does not scale across oceans, or gave up global ordering.

The idea: measure the clock error, then wait it out

Spanner’s answer is TrueTime, an API that refuses to pretend. Instead of a timestamp it returns an interval, [earliest, latest], that is guaranteed to contain the true time; the width of the interval, 2ε, is the uncertainty, and the system knows it at every moment. Behind the API are GPS receivers and atomic clocks in every data centre and a daemon on every machine that polls them and widens its interval between polls to cover drift. In the paper ε is typically between one and seven milliseconds.

How it works

Data is split into tablets, and each tablet is replicated across zones by its own Paxos group; a write is durable on a majority before the leader acknowledges it. A transaction that touches several tablets runs two-phase commit between their Paxos leaders, with the participants themselves replicated, so the classic weakness of two-phase commit, a coordinator that dies and leaves everyone waiting, cannot happen. Locks are held in the leaders; this is two-phase locking inside Paxos inside two-phase commit, and the paper is matter-of-fact about stacking them.

The payoff is reads. Because timestamps mean the same thing everywhere, Spanner can serve a snapshot read at any past timestamp from any replica that is caught up to it, without locks and without talking to the leader. A query over the whole database at “ten seconds ago” is consistent across every continent. Even schema changes become transactions with a timestamp in the future, applied simultaneously worldwide when the clock passes it.

What it changed

Spanner ended the assumption that global scale required giving up transactions, and started the category now called NewSQL. CockroachDB and YugabyteDB reproduce its architecture without special hardware, using hybrid logical clocks and accepting slightly weaker guarantees; TiDB takes a similar route. Google shipped Spanner itself as Cloud Spanner in 2017, and in 2024 Amazon’s Aurora DSQL adopted the same bounded-time approach using precise clock synchronisation. Brewer’s follow-up essay argues that Spanner is “effectively CA”: it is technically CP, but Google’s private network makes partitions so rare that users never see the difference.

Where you meet it today

• Cloud Spanner, CockroachDB, YugabyteDB, TiDB. If a database advertises global strong consistency, it is some variation of this paper.
• Bounded clocks as a service. AWS Time Sync and the ClockBound library expose an interval, not a number, so applications can play Spanner’s trick themselves.
• Snapshot reads at a timestamp. “As of” queries and time-travel reads in modern databases descend from Spanner’s read-only transactions.
• The design principle. When you cannot eliminate uncertainty, measure it and wait it out; the same idea reappears in leases, fencing tokens and lock expiry.