Reinforcement Learning

01Overview

In the 1950s optimal control and dynamic programming supplied the theory; in 1989 Watkins proposed Q-learning, estimating action values by iterating the Bellman equation. In 1992 TD-Gammon reached expert backgammon through self-play, showing what temporal-difference learning could do.

In 2013 DeepMind’s deep Q-network (DQN) learned Atari games straight from pixels; in 2016 AlphaGo combined policy and value networks with Monte Carlo tree search to beat Lee Sedol; in 2017 AlphaZero learned Go, chess and shogi from scratch by self-play. Sample efficiency and reward design remain hard, but these results carried reinforcement learning into robotics, recommendation and reasoning models.

02History

1989
Q-learning
Model-free temporal-difference control.
1992
TD-Gammon
An early success at self-play.
2013
DQN
Deep reinforcement learning plays Atari.
2016
AlphaGo
It beats Lee Sedol.
2017
AlphaZero
General game learning from scratch.

03Core concepts

Policy

The rule mapping states to actions that the agent improves.

Temporal difference

Update an estimate from the difference between successive predictions.

Exploration and exploitation

Try new actions, or use the best known one — a permanent tension.

04Hands-on

Do it once — it beats reading it ten times

Implement Q-learning on a 5×5 grid world and plot the learned policy after 1000 episodes

Tick this once you have built it and written down what you learned; the topic is then marked as built.

05Self-check

Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.

Can you explain "Policy"?

The rule mapping states to actions that the agent improves.

Can you explain "Temporal difference"?

Update an estimate from the difference between successive predictions.

Can you explain "Exploration and exploitation"?

Try new actions, or use the best known one — a permanent tension.

What problem did it solve, and why was there nothing better at the time?

Go back to the history section: how was this done before, and what did it cost?

If you had to build it today, how would you break it down?

Use the hands-on task above: write a plan of at most three steps before you start.

06Key people

Richard Sutton

1956 –

Temporal-difference learning and the textbook on RL.

David Silver

1976 –

Lead researcher on AlphaGo and AlphaZero.

07Further reading

Reinforcement Learning: An IntroductionSutton & Barto · 1998
Human-level control through deep reinforcement learningMnih et al. · 2015