01Overview
In 1654 Pascal and Fermat settled the problem of points in their letters, and probability was on its way. Jacob Bernoulli proved the law of large numbers: observed frequencies settle near the probability. Laplace applied probability to celestial mechanics and to evidence, and Bayes gave the rule for updating belief in the light of new evidence.
In 1933 Kolmogorov put probability on measure-theoretic axioms and made it a rigorous branch of mathematics. Statistical inference, Markov chain Monte Carlo and Bayesian networks matured in the second half of the century. Today, from A/B tests to machine learning, probability is the most general language we have for saying “I do not know”.
02History
- 1654
Pascal and Fermat correspond
Probability is born from a problem about gambling.
- 1713
Bernoulli, Ars Conjectandi
The law of large numbers is proved for the first time.
- 1763
Bayes' theorem published
Inverse probability and the updating of belief.
- 1933
Kolmogorov's axiomatisation
Probability becomes part of measure theory.
- 1990
MCMC becomes practical
High-dimensional posterior inference becomes feasible.
03Core concepts
Law of large numbers
As the sample grows, the sample mean converges to the expectation.
Bayesian inference
Prior times likelihood gives the posterior, updated as evidence arrives.
Central limit theorem
The sum of many independent random variables is approximately normal.
04Hands-on
Do it once — it beats reading it ten times
Verify the central limit theorem with 100,000 simulations: sum uniform variables, plot the histogram, compare with the normal curve
05Self-check
Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.
Can you explain "Law of large numbers"?
As the sample grows, the sample mean converges to the expectation.
Can you explain "Bayesian inference"?
Prior times likelihood gives the posterior, updated as evidence arrives.
Can you explain "Central limit theorem"?
The sum of many independent random variables is approximately normal.
What problem did it solve, and why was there nothing better at the time?
Go back to the history section: how was this done before, and what did it cost?
If you had to build it today, how would you break it down?
Use the hands-on task above: write a plan of at most three steps before you start.
06Key people
Blaise Pascal
1623 – 1662
A pioneer of probability and of the calculating machine.
Andrey Kolmogorov
1903 – 1987
Built the axiomatic foundation of probability.
07Further reading
- Ars Conjectandi
- Foundations of the Theory of Probability