Information theory quantifies how much uncertainty is reduced when you learn something.
For example, if you start with 16 equally likely options, your total uncertainty is bits.
Each bit of information halves the number of remaining possibilities.
- After learning something that narrows 16 → 8 options, you gain bit.
- Another clue that shrinks 8 → 4 gives another bit.
Information adds up: total gained is equal to bits, leaving bits of uncertainty.
Formula:
Each bit = one “yes/no” question that halves the search space.