01Overview
In 1999 NVIDIA’s GeForce 256 was first called a GPU: it fixed transform, lighting and rasterisation onto a massively parallel pipeline. From 2001 programmable shaders let developers write small programs running on thousands of cores.
In 2006 CUDA turned the GPU into a general parallel processor. In 2012 AlexNet trained a model on two GTX 580s that stunned ImageNet, and the GPU became the infrastructure of AI. Today GPUs, TPUs and NPUs form heterogeneous computing together: the CPU controls, the accelerators deliver throughput.
02History
- 1999
GeForce 256
The word GPU reaches a wide audience.
- 2001
Programmable shaders
GPUs begin to accept general programs.
- 2006
CUDA
An operating system for general-purpose GPU computing.
- 2012
AlexNet
GPU-trained deep networks begin.
- 2017
TPUs and dedicated accelerators
Google puts tensor arithmetic into silicon.
03Core concepts
SIMT
Single instruction, multiple threads: one instruction drives many threads.
Heterogeneous computing
CPU and accelerators split the work each is good at.
Memory bandwidth
The key constraint on GPU performance, often scarcer than raw compute.
04Hands-on
Do it once — it beats reading it ten times
Write a CUDA kernel that adds two 1M-element arrays and compare its time with the CPU version
05Self-check
Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.
Can you explain "SIMT"?
Single instruction, multiple threads: one instruction drives many threads.
Can you explain "Heterogeneous computing"?
CPU and accelerators split the work each is good at.
Can you explain "Memory bandwidth"?
The key constraint on GPU performance, often scarcer than raw compute.
What problem did it solve, and why was there nothing better at the time?
Go back to the history section: how was this done before, and what did it cost?
If you had to build it today, how would you break it down?
Use the hands-on task above: write a plan of at most three steps before you start.
06Key people
Jensen Huang
1963 –
NVIDIA co-founder; pushed general-purpose GPU computing.
Ian Buck
1970 –
Early leader of CUDA.
07Further reading
- CUDA Programming Guide
- Computer Architecture: A Quantitative Approach