01Overview

Google published GFS in 2003 and MapReduce in 2004: split the computation into map and reduce stages and let the framework handle scheduling, fault tolerance and data locality. In 2006 Hadoop open-sourced the idea, and with HDFS it became the first generation of big-data infrastructure.

The batch model was simple but slow, especially for iterative algorithms. In 2010 Spark used in-memory resilient distributed datasets (RDDs) to speed iteration by an order of magnitude, and Flink treated streaming as a first-class citizen. Today the boundary of “big data” has blurred, and batch-plus-streaming plus the lakehouse have become the new consensus.

02History

  1. 2003

    The GFS paper

    A distributed file system.

  2. 2004

    The MapReduce paper

    A programming model for batch processing.

  3. 2006

    Hadoop

    The open-source implementation.

  4. 2010

    Spark

    In-memory computing and RDDs.

  5. 2014

    Flink

    Stream and batch in one engine.

03Core concepts

Data locality

Schedule computation on the machine that holds the data to avoid network transfer.

Sharding

Cut data into blocks and spread them across machines.

Fault tolerance

Re-run the failed task rather than restarting the whole job.

04Hands-on

Do it once — it beats reading it ten times

Run word count on a 1 GB file with MapReduce and with Spark; compare wall-clock time and bytes shuffled

05Self-check

Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.

Can you explain "Data locality"?

Schedule computation on the machine that holds the data to avoid network transfer.

Can you explain "Sharding"?

Cut data into blocks and spread them across machines.

Can you explain "Fault tolerance"?

Re-run the failed task rather than restarting the whole job.

What problem did it solve, and why was there nothing better at the time?

Go back to the history section: how was this done before, and what did it cost?

If you had to build it today, how would you break it down?

Use the hands-on task above: write a plan of at most three steps before you start.

06Key people

J

Jeff Dean

1968 –

Core author of GFS, MapReduce and BigTable.

M

Matei Zaharia

1984 –

Creator of Spark.

07Further reading

  • MapReduce: Simplified Data Processing on Large ClustersDean & Ghemawat · 2004
  • Spark: The Definitive GuideDamji et al. · 2017