Differential Entropy — Mathematics Knowledge System

Overview

Differential entropy h(X) is the continuous analog of Shannon entropy for a continuous random variable X with probability density function f(x). It is defined as h(X) = -∫ f(x) log f(x) dx. Unlike discrete entropy, differential entropy can be negative, is not invariant under change of variables, and is not an absolute measure of information. However, differences of differential entropies (such as mutual information for continuous variables) are well-behaved and have the same interpretations as in the discrete case. The maximum differential entropy for a fixed variance is achieved by the Gaussian distribution — a key result in information theory and signal processing.

Intuition

For a continuous random variable, we cannot ask 'what is the probability of exactly X = 3.14159?' because individual points have probability 0. Differential entropy extends the entropy idea by using the probability density. For a uniform distribution on [0,1], h(X) = 0 bits; for a uniform distribution on [0,2], h(X) = 1 bit — wider spread means more uncertainty. For a very tight Gaussian (small variance), h is large and negative; for a wide Gaussian, h is large and positive. Crucially, h(X) alone is not meaningful as an absolute quantity — only differences like I(X;Y) = h(X) - h(X|Y) are truly meaningful.

Formal Definition

Definition

For a continuous random variable X with density f(x), the differential entropy is:

h(X) = -\int_{-\infty}^{\infty} f(x) \log f(x)\, dx

Differential entropy

h(X) = -\mathbb{E}[\log f(X)]

Expectation form

h(X + c) = h(X)

Translation invariance

h(aX) = h(X) + \log|a|

Scaling property

h(\mathcal{N}(\mu, \sigma^2)) = \tfrac{1}{2}\log(2\pi e\, \sigma^2)

Gaussian differential entropy (nats, divide by ln 2 for bits)

Notation

Notation	Meaning
$h(X)$	Differential entropy of continuous random variable X
$f(x)$	Probability density function of X
$h(X\|Y)$	Conditional differential entropy

Theorems

Theorem 1: Maximum Entropy Theorem (Gaussian Maximizes Entropy)

Among all continuous distributions on \mathbb{R} with fixed variance \sigma^2, the Gaussian \mathcal{N}(\mu, \sigma^2) has the maximum differential entropy h(X) = \frac{1}{2}\log_2(2\pi e\, \sigma^2) \text{ bits}.

Theorem 2: Relation to Discrete Entropy via Quantization

If X is quantized to resolution \Delta, then H(X_\Delta) \approx h(X) - \log_2 \Delta as \Delta \to 0, where H(X_\Delta) is the discrete entropy of the quantized variable.

Theorem 3: Differential Entropy Under Invertible Transforms

For a differentiable invertible function g and Y = g(X): h(Y) = h(X) + \mathbb{E}[\log |g'(X)|].

Worked Examples

1
The density is f(x) = 1/a for x ∈ [0,a], 0 otherwise.
$f(x) = 1/a,\ x \in [0,a]$
2
Apply the definition.
$h(X) = -\int_0^a \frac{1}{a}\log_2\frac{1}{a}\,dx$
3
The integrand is constant.
$h(X) = -\frac{1}{a}\log_2\frac{1}{a} \cdot a = \log_2 a$

✓ Answer

h(Uniform[0,a]) = log₂ a bits. For a=1, h=0. For a=2, h=1 bit. Can be negative if a < 1.

Practice Problems

Easyfill in blank

X ~ Exponential(λ) with density f(x) = λe^{-λx} for x ≥ 0. Compute h(X) in nats.

Mediumfree response

Why can differential entropy be negative, and give an example where h(X) < 0?

Common Mistakes

Common Mistake

Thinking differential entropy h(X) has the same absolute meaning as discrete entropy H(X).

Differential entropy is not invariant under invertible transforms (unlike discrete entropy) and can be negative. Only differences like h(X) - h(X|Y) = I(X;Y) are invariant and meaningful.

Common Mistake

Assuming h(X) ≥ 0 always, by analogy with discrete entropy.

h(X) can be negative. For example, h(Uniform[0, 0.5]) = log₂(0.5) = -1 bit. Discrete entropy H(X) ≥ 0 always, but differential entropy has no such bound.

Quiz

Historical Background

Shannon briefly defined differential entropy in his 1948 paper as the natural extension of discrete entropy to continuous distributions, used to analyze the Gaussian channel. The subtleties of differential entropy — its dependence on coordinate system, possible negativity, and non-invariance — were discussed extensively by later researchers. The maximum entropy result for Gaussian distributions had deep implications for channel capacity and the design of communication systems.

1948
Shannon defines differential entropy and uses it to analyze Gaussian channels.
Claude Shannon
1950s
Differential entropy properties (coordinate dependence, maximum entropy) systematically studied.
1990s
Differential entropy becomes fundamental to independent component analysis (ICA) in machine learning.
Aapo Hyvarinen

Summary

Differential entropy h(X) = -∫ f(x) log f(x) dx extends Shannon entropy to continuous random variables with density f.
Unlike discrete entropy, h(X) can be negative, is not invariant under invertible transforms, and has no absolute interpretation.
The Gaussian maximizes h(X) among all distributions with fixed variance σ²: h(N(μ,σ²)) = (1/2) log₂(2πeσ²) bits.
Meaningful quantities involving h are differences: mutual information I(X;Y) = h(X) - h(X|Y) ≥ 0 is invariant and well-defined.

References

PaperShannon, C.E. (1948). A Mathematical Theory of Communication. Bell System Technical Journal.
BookCover, T. and Thomas, J. (2006). Elements of Information Theory, 2nd ed. Wiley. Ch. 8.
WebsiteWikipedia — Differential entropy