How Shannon Entropy Quantifies Information and Surprise
This article explores the mathematical concept of Shannon entropy and how it measures the average amount of information or surprise contained within a random variable. We will break down the foundational relationship between probability and "surprisal," explain the mathematical formula introduced by Claude Shannon in 1948, and illustrate how different probability distributions directly dictate the entropy of a system using practical examples.
The Core Concept: Information as Surprise
In mathematics and information theory, the amount of "information" gained from an event is directly tied to how surprising that event is. Surprise is inversely related to probability:
- High Probability = Low Surprise: If the sun rises tomorrow, you gain almost no new information because the event was practically guaranteed.
- Low Probability = High Surprise: If it snows in the Sahara Desert, the event is highly improbable, meaning it carries a massive amount of surprise and, therefore, a high amount of information.
Before calculating the average surprise of a whole system (the random variable), we must first quantify the surprise of a single outcome. Mathematicians call this surprisal or self-information, denoted as \(I(x)\).
For an outcome \(x\) with a probability \(P(x)\), the surprisal is calculated as:
\[I(x) = -\log_2(P(x))\]
Using a base-2 logarithm means the resulting unit of information is measured in bits (binary digits). Because probabilities are always between 0 and 1, the logarithm of a probability is always negative or zero. The negative sign in the formula ensures that information is represented as a positive number.
If an event is certain to happen (\(P(x) = 1\)), the formula yields \(-\log_2(1) = 0\) bits of surprise. If an event has a 25% chance of happening (\(P(x) = 0.25\)), the surprise is \(-\log_2(0.25) = 2\) bits.
Moving from Single Events to Random Variables
A random variable, denoted as \(X\), is not a single event; it is a set of all possible outcomes of a random process, each with its own probability. Because you do not know which outcome will occur next, you cannot use the surprisal of just one outcome to describe the whole variable. Instead, you must calculate the average amount of surprise you expect to get over many trials.
In statistics, the average of a random variable is called its "expected value." You find the expected value by multiplying each possible outcome by its probability and adding them all together.
Shannon entropy, denoted as \(H(X)\), is simply the expected value of the surprisal across all possible outcomes of the random variable \(X\).
The Shannon Entropy Formula
For a discrete random variable \(X\) with possible outcomes \(x_1, x_2, ..., x_n\), the Shannon entropy is defined as:
\[H(X) = -\sum P(x) \log_2 P(x)\]
Here is how the formula works step-by-step:
- \(P(x)\): Find the probability of a specific outcome.
- \(\log_2 P(x)\): Calculate the negative surprisal of that specific outcome.
- \(P(x) \log_2 P(x)\): Weight the surprise of that outcome by how often it actually occurs.
- \(-\sum\): Sum these weighted values together for every possible outcome, and flip the sign to make the final result positive.
The result, \(H(X)\), tells you the average number of bits of information generated every time the random variable produces an outcome.
A Practical Example: Flipping Coins
To see how Shannon entropy quantifies uncertainty and surprise, consider three different coins representing three different random variables.
1. The Fair Coin (Maximum Entropy) A fair coin has two outcomes: Heads (\(P = 0.5\)) and Tails (\(P = 0.5\)).
- Surprise of Heads: \(-\log_2(0.5) = 1\) bit.
- Surprise of Tails: \(-\log_2(0.5) = 1\) bit.
- Average Surprise (Entropy): \((0.5 \times 1) + (0.5 \times 1) = 1\) bit.
Because both outcomes are equally likely, uncertainty is at its absolute maximum. Every flip gives you an average of 1 bit of information.
2. The Biased Coin (Low Entropy) Imagine a weighted coin that lands on Heads 90% of the time (\(P = 0.9\)) and Tails 10% of the time (\(P = 0.1\)).
- Surprise of Heads: \(-\log_2(0.9) \approx 0.15\) bits.
- Surprise of Tails: \(-\log_2(0.1) \approx 3.32\) bits.
- Average Surprise (Entropy): \((0.9 \times 0.15) + (0.1 \times 3.32) \approx 0.135 + 0.332 = 0.47\) bits.
Even though landing on Tails is very surprising (3.32 bits), it rarely happens. Most of the time, you get the unsurprising Heads (0.15 bits). Therefore, the average surprise of this coin drops to 0.47 bits. It contains less information than a fair coin because it is more predictable.
3. The Trick Coin (Zero Entropy) A coin with two heads lands on Heads 100% of the time (\(P = 1.0\)).
- Surprise of Heads: \(-\log_2(1.0) = 0\) bits.
- Average Surprise (Entropy): \(1.0 \times 0 = 0\) bits.
There is no randomness here. Because the outcome is predetermined, there is zero surprise, and flipping the coin provides 0 bits of information.
Summary of Properties
Through this mathematical framework, Shannon entropy establishes three fundamental rules about information in random variables:
- Uniformity maximizes entropy: A random variable has the highest possible average surprise when all of its outcomes are equally likely.
- Predictability reduces entropy: As one outcome becomes more probable than others, the average surprise decreases.
- Certainty eliminates entropy: When an outcome has a probability of 1, the entropy is 0, meaning the variable contains no information.