01Overview

Distributed databases were theorised in the 1980s but limited by networks and hardware. In 2000 the CAP theorem reminded everyone that under a network partition you cannot have both consistency and availability. In 2012 Google’s Spanner used TrueTime — GPS and atomic clocks — to make clock uncertainty explicit and delivered strongly consistent transactions worldwide.

After Spanner, NewSQL systems such as CockroachDB, TiDB and YugabyteDB replicated logs with Raft and Paxos, folding sharding, transactions and consensus into one system. Their aim is not to replace every database but to offer a simpler answer than hand-rolled sharding wherever global distribution and strong consistency are required.

02History

  1. 2000

    The CAP theorem

    Consistency, availability, partition tolerance.

  2. 2007

    Dynamo

    Eventual consistency and consistent hashing.

  3. 2012

    Spanner

    Globally distributed, strongly consistent transactions.

  4. 2015

    Raft becomes popular

    A consensus algorithm you can understand.

  5. 2017

    CockroachDB and TiDB

    NewSQL goes commercial.

03Core concepts

Sharding

Split data across nodes by range or by hash.

Replication

Keep the same data on several nodes to raise availability.

CAP

Consistency, availability and partition tolerance: you cannot have all three.

04Hands-on

Do it once — it beats reading it ten times

Split a small table across three nodes by hash, then run a cross-shard join by hand and count the round trips

05Self-check

Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.

Can you explain "Sharding"?

Split data across nodes by range or by hash.

Can you explain "Replication"?

Keep the same data on several nodes to raise availability.

Can you explain "CAP"?

Consistency, availability and partition tolerance: you cannot have all three.

What problem did it solve, and why was there nothing better at the time?

Go back to the history section: how was this done before, and what did it cost?

If you had to build it today, how would you break it down?

Use the hands-on task above: write a plan of at most three steps before you start.

06Key people

E

Eric Brewer

1964 –

Proposed the CAP theorem.

J

James Corbett

1968 –

Lead author of the Spanner paper.

07Further reading

  • Spanner: Google’s Globally-Distributed DatabaseCorbett et al. · 2012
  • Designing Data-Intensive ApplicationsMartin Kleppmann · 2018