01Overview

In 1986 Rumelhart, Hinton and Williams popularised backpropagation, which made multi-layer networks trainable. In 1989 LeCun used a convolutional network to read handwritten digits, deployed on bank cheques. But limited data and compute sent neural networks quiet again in the late 1990s.

In 2006 Hinton’s layer-wise pretraining of deep belief networks revived “deep”; in 2012 AlexNet cut the ImageNet error rate from 26% to 15%, with GPUs, ReLU and dropout all contributing. ResNet (2015), batch normalisation and attention kept pushing the numbers down. The essence of deep learning is representation learning: instead of hand-designed features, the network discovers its own hierarchy.

02History

  1. 1986

    Backpropagation

    Multi-layer networks become trainable.

  2. 1989

    LeNet

    A convolutional network reads handwritten digits.

  3. 2006

    Deep belief networks

    Layer-by-layer pretraining.

  4. 2012

    AlexNet

    GPU-trained deep networks begin.

  5. 2015

    ResNet

    Residual connections train very deep networks.

03Core concepts

Backpropagation

Compute gradients with the chain rule, from the output layer back to the input.

Representation learning

Let the model learn features rather than relying on hand engineering.

Overfitting

Memorising the training data while failing to generalise.

04Hands-on

Do it once — it beats reading it ten times

Train a tiny CNN on MNIST, then remove the convolutions and compare accuracy and parameter count

05Self-check

Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.

Can you explain "Backpropagation"?

Compute gradients with the chain rule, from the output layer back to the input.

Can you explain "Representation learning"?

Let the model learn features rather than relying on hand engineering.

Can you explain "Overfitting"?

Memorising the training data while failing to generalise.

What problem did it solve, and why was there nothing better at the time?

Go back to the history section: how was this done before, and what did it cost?

If you had to build it today, how would you break it down?

Use the hands-on task above: write a plan of at most three steps before you start.

06Key people

G

Geoffrey Hinton

1947 –

A driving force behind backpropagation and deep learning.

Y

Yann LeCun

1960 –

A founder of convolutional networks.

07Further reading

  • Deep LearningGoodfellow, Bengio & Courville · 2016
  • ImageNet Classification with Deep Convolutional Neural NetworksKrizhevsky et al. · 2012