Transformers & Large Language Models

01Overview

In 2014 Bahdanau and colleagues introduced attention into machine translation; in 2017 Google’s Attention Is All You Need proposed the Transformer: self-attention plus feed-forward layers, no recurrence, and training that parallelises heavily. Its core is that every position can attend directly to every other.

In 2018 BERT pretrained a bidirectional encoder and GPT an autoregressive decoder. In 2020 GPT-3’s 175 billion parameters demonstrated few-shot learning; in 2022 ChatGPT put a conversational model in front of the public. Transformers spread from language to vision (ViT), proteins (AlphaFold) and multimodality. The price is quadratic attention, and sparsity plus linear attention are still being explored.

02History

2014
Attention
Bahdanau applies it to machine translation.
2017
Transformer
The self-attention architecture.
2018
BERT and GPT
Pretrain then fine-tune; pretrain then generate.
2020
GPT-3
Scale brings few-shot ability.
2022
ChatGPT
Large models reach the mass market.

03Core concepts

Self-attention

Each position aggregates information from others according to relevance.

Positional encoding

Inject order information into a structure with no recurrence.

Scaling laws

Loss falls as a power law in parameters, data and compute.

04Hands-on

Do it once — it beats reading it ten times

Implement single-head self-attention in 20 lines of NumPy and verify the output shape and the causal mask

Tick this once you have built it and written down what you learned; the topic is then marked as built.

05Self-check

Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.

Can you explain "Self-attention"?

Each position aggregates information from others according to relevance.

Can you explain "Positional encoding"?

Inject order information into a structure with no recurrence.

Can you explain "Scaling laws"?

Loss falls as a power law in parameters, data and compute.

What problem did it solve, and why was there nothing better at the time?

Go back to the history section: how was this done before, and what did it cost?

If you had to build it today, how would you break it down?

Use the hands-on task above: write a plan of at most three steps before you start.

06Key people

Ashish Vaswani

1986 –

Lead author of the Transformer paper.

Ilya Sutskever

1986 –

OpenAI co-founder; a driving force behind the GPT series.

07Further reading

Attention Is All You NeedVaswani et al. · 2017
Language Models are Few-Shot LearnersBrown et al. · 2020