01Overview
In the 1980s the limits of expert systems turned researchers to data. In 1988 Pearl’s Probabilistic Reasoning in Intelligent Systems brought Bayesian networks into AI; in 1995 Cortes and Vapnik introduced support vector machines, handling non-linear classification with kernels and offering theoretical generalisation guarantees.
The keyword of the period was generalisation: a model must not merely fit the training data but perform on unseen data. Cross-validation, regularisation and the bias–variance trade-off became standard tools. Random forests (2001) and gradient-boosted trees then dominated tabular data for years. Deep learning did not replace them; it added representation learning on top.
02History
- 1988
Bayesian networks
Probabilistic graphical models.
- 1995
Support vector machines
Kernels and the maximum margin.
- 2001
Random forests
The face of ensemble learning.
- 2000
Gradient-boosted trees
The workhorse for tabular data.
03Core concepts
Generalisation
Performance on unseen data rather than on training error.
Regularisation
Constrain model complexity to prevent overfitting.
Kernel methods
Map samples into a higher-dimensional space to make non-linear problems separable.
04Hands-on
Do it once — it beats reading it ten times
Train a decision tree and an SVM on the same small dataset; plot the bias–variance difference as you change the data size
05Self-check
Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.
Can you explain "Generalisation"?
Performance on unseen data rather than on training error.
Can you explain "Regularisation"?
Constrain model complexity to prevent overfitting.
Can you explain "Kernel methods"?
Map samples into a higher-dimensional space to make non-linear problems separable.
What problem did it solve, and why was there nothing better at the time?
Go back to the history section: how was this done before, and what did it cost?
If you had to build it today, how would you break it down?
Use the hands-on task above: write a plan of at most three steps before you start.
06Key people
Vladimir Vapnik
1936 –
Statistical learning theory and support vector machines.
Judea Pearl
1936 –
Bayesian networks and causal inference.
07Further reading
- The Elements of Statistical Learning
- Pattern Recognition and Machine Learning