Shannon Entropy — Mathematics Knowledge System

Overview

Shannon entropy H(X) quantifies the average uncertainty or information content of a random variable X. For a discrete distribution over n outcomes with probabilities p₁, …, pₙ, it is defined as H(X) = -∑ pᵢ log pᵢ. Choosing log base 2 gives units of bits; natural log gives nats; log base 10 gives Hartleys (or dits). Entropy is maximized when all outcomes are equally likely and equals zero when the outcome is certain. It is the foundational quantity of information theory, underpinning lossless compression limits, channel capacity, and the thermodynamic analogy between statistical physics and communication.

Intuition

Entropy measures surprise on average. If you flip a fair coin you are maximally uncertain — either outcome is equally likely, so you gain 1 bit of information per flip. If the coin is biased 99:1, most flips are predictable (heads almost always) and the average information per flip is much less than 1 bit. Entropy captures this: uniform distributions are maximally entropic (maximum surprise); degenerate distributions (one outcome certain) have zero entropy (no surprise). Think of it as the average number of yes/no questions needed to identify the outcome of the random variable.

Formal Definition

Definition

For a discrete random variable X taking values x₁, …, xₙ with probabilities pᵢ = P(X = xᵢ), the Shannon entropy in bits is:

H(X) = -\sum_{i=1}^{n} p_i \log_2 p_i

Shannon entropy (bits)

H(X) = -\sum_{i=1}^{n} p_i \ln p_i

Shannon entropy (nats)

H(X) = \mathbb{E}\left[-\log_2 p(X)\right]

Entropy as expected surprise

0 \leq H(X) \leq \log_2 n

Bounds on entropy

H(X) = 0 \iff p_k = 1 \text{ for some } k

Zero entropy iff deterministic

Notation

Notation	Meaning
$H(X)$	Shannon entropy of random variable X
$H(\mathbf{p})$	Entropy of probability vector p
$H(X\|Y)$	Conditional entropy of X given Y
$H(X,Y)$	Joint entropy of X and Y

Theorems

Theorem 1: Non-negativity and Maximum Entropy

For any discrete random variable X with n outcomes, 0 \leq H(X) \leq \log_2 n, with H(X) = 0 iff X is deterministic and H(X) = \log_2 n iff X is uniformly distributed.

Theorem 2: Chain Rule for Entropy

H(X_1, X_2, \ldots, X_n) = \sum_{i=1}^{n} H(X_i \mid X_1, \ldots, X_{i-1})

Theorem 3: Subadditivity of Entropy

H(X, Y) \leq H(X) + H(Y), with equality iff X and Y are independent.

Worked Examples

1
A fair coin has p(heads) = p(tails) = 1/2.
$p_1 = p_2 = \tfrac{1}{2}$
2
Apply the entropy formula.
$H(X) = -\tfrac{1}{2}\log_2\tfrac{1}{2} - \tfrac{1}{2}\log_2\tfrac{1}{2}$
3
Since log₂(1/2) = -1, each term contributes 1/2 × 1 = 1/2.
$H(X) = \tfrac{1}{2}(1) + \tfrac{1}{2}(1) = 1 \text{ bit}$

✓ Answer

H(X) = 1 bit — the maximum entropy for a two-outcome variable.

Practice Problems

Easyfill in blank

A loaded die shows face 6 with probability 1/2 and all other faces with probability 1/10 each. What is H(X) in bits (to 2 decimal places)?

Mediumproof writing

Prove that H(X) ≤ log₂ n for any discrete random variable with n outcomes, with equality iff X is uniform.

Common Mistakes

Common Mistake

Treating 0·log(0) as undefined and excluding those terms.

By convention, 0·log 0 = 0 (using the limit limₓ→₀ x log x = 0). Outcomes with probability 0 contribute 0 to entropy.

Common Mistake

Confusing entropy with variance — a high-variance distribution need not have high entropy.

Entropy depends only on the probability values, not the actual outcome values. Two distributions with the same probability vector have the same entropy regardless of their support.

Quiz

Historical Background

Claude Shannon introduced entropy in his landmark 1948 paper 'A Mathematical Theory of Communication,' published in the Bell System Technical Journal. Shannon borrowed the term from Ludwig Boltzmann's statistical mechanics entropy, observing a mathematical parallel between thermodynamic uncertainty and communication uncertainty. John von Neumann reportedly suggested the word 'entropy' to Shannon since 'no one knows what entropy really is.' Shannon's formulation unified disparate results in telegraphy, cryptography, and statistics into a single coherent theory.

1872
Boltzmann introduces H-theorem and statistical entropy in thermodynamics.
Ludwig Boltzmann
1948
Shannon publishes 'A Mathematical Theory of Communication', defining entropy for discrete sources.
Claude Shannon
1949
Shannon and Weaver publish the book 'The Mathematical Theory of Communication'.
Claude Shannon, Warren Weaver
1951
Shannon proves the noiseless coding theorem, linking entropy to optimal compression rates.
Claude Shannon

Summary

Shannon entropy H(X) = -∑ pᵢ log pᵢ measures average uncertainty in bits (log base 2), nats (natural log), or Hartleys (log base 10).
Entropy is non-negative, equals 0 for deterministic variables, and is maximized (log₂ n bits) for the uniform distribution over n outcomes.
The chain rule H(X,Y) = H(X) + H(Y|X) decomposes joint entropy; subadditivity H(X,Y) ≤ H(X)+H(Y) holds with equality iff X,Y are independent.
Shannon's source coding theorem states that the minimum average code length for lossless compression is H(X) bits per symbol.

References

PaperShannon, C.E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal, 27, 379-423.
BookCover, T. and Thomas, J. (2006). Elements of Information Theory, 2nd ed. Wiley.
WebsiteWikipedia — Entropy (information theory)