01Overview
In 1948 Shannon argued in A Mathematical Theory of Communication that the amount of information has nothing to do with content and everything to do with uncertainty. A fair coin carries one bit because it has two equally likely outcomes; the entropy of a text depends on how often each character occurs. That definition gave compression, error correction and cryptography one common measure.
Shannon also proved that below channel capacity some coding scheme makes the error probability arbitrarily small, and above it none exists. The theorem named no particular code, yet it set the target for seventy years of coding theory. From LDPC codes on disks to Polar codes in 5G, engineering keeps closing in on that limit.
02History
- 1928
Hartley measures information in logarithms
A precursor to Shannon's theory.
- 1948
A Mathematical Theory of Communication
Entropy, channel capacity and rate–distortion arrive together.
- 1950
Hamming codes
The first practical error-correcting code.
- 1993
Turbo codes
Iterative decoding that approaches the Shannon limit.
- 2008
Polar codes
Adopted for 5G control channels.
03Core concepts
Entropy
A measure of uncertainty, and the theoretical lower bound on lossless compression.
Channel capacity
The maximum rate at which information can cross a noisy channel reliably.
Error-correcting codes
Add redundancy so the receiver can detect and repair transmission errors.
04Hands-on
Do it once — it beats reading it ten times
Compute the character entropy of a 1 KB text, compare it with the size after gzip, and explain the gap
05Self-check
Say it in your own words first, then open the answer. Whatever you cannot explain is what to review next.
Can you explain "Entropy"?
A measure of uncertainty, and the theoretical lower bound on lossless compression.
Can you explain "Channel capacity"?
The maximum rate at which information can cross a noisy channel reliably.
Can you explain "Error-correcting codes"?
Add redundancy so the receiver can detect and repair transmission errors.
What problem did it solve, and why was there nothing better at the time?
Go back to the history section: how was this done before, and what did it cost?
If you had to build it today, how would you break it down?
Use the hands-on task above: write a plan of at most three steps before you start.
06Key people
Claude Shannon
1916 – 2001
Founder of information theory and of digital circuit design.
Richard Hamming
1915 – 1998
Invented Hamming codes and the Hamming distance.
07Further reading
- A Mathematical Theory of Communication
- Elements of Information Theory