Information Geometry — Mathematics Knowledge System

Overview

Information geometry applies differential geometry to the space of probability distributions, treating families of distributions as smooth manifolds. The Fisher information matrix defines a Riemannian metric (Fisher-Rao metric) on this statistical manifold. The KL divergence plays the role of a (asymmetric) distance, and its symmetrisation gives a metric. The theory reveals the geometric structure underlying statistical inference: the natural gradient, exponential families as flat submanifolds, and the duality between e-flat and m-flat submanifolds. Applications include neural network optimisation (natural gradient descent), statistical physics, and quantum information.

Intuition

Think of the space of all Gaussian distributions parametrised by (mu, sigma): it forms a 2D manifold. Different directions in this manifold correspond to different ways of changing the distribution -- changing the mean vs. changing the variance. The Fisher information metric tells you the 'natural' notion of distance between nearby distributions: two distributions that are 'statistically indistinguishable' (close in KL divergence) are metrically close. The natural gradient descent moves in the direction of steepest ascent in this Riemannian manifold, not the flat Euclidean parameter space.

Formal Definition

Definition

Let {p(x; theta) : theta in Theta} be a parametric family of distributions on X. The Fisher information matrix g_{ij}(theta) = E[d/d theta_i log p * d/d theta_j log p] defines a Riemannian metric on Theta (the Fisher-Rao metric). The Amari alpha-connection defines a family of affine connections on the statistical manifold. For alpha = 0, this is the Levi-Civita connection of the Fisher metric; for alpha = +-1, the e-connection and m-connection -- dual affine connections. Exponential families are e-flat (zero curvature under e-connection).

g_{ij}(\theta) = \mathbb{E}_{p_\theta}\!\left[\frac{\partial \log p}{\partial \theta_i}\frac{\partial \log p}{\partial \theta_j}\right]

Fisher information matrix (Riemannian metric)

D_{\mathrm{KL}}(p \| q) = \int p(x) \log\frac{p(x)}{q(x)}\, dx

KL divergence as potential for geometry

\tilde{\nabla} L(\theta) = G(\theta)^{-1} \nabla L(\theta)

Natural gradient

p(x;\theta) = h(x) \exp(\theta^T T(x) - A(\theta))

Exponential family (e-flat manifold)

Notation

Notation	Meaning
$g_{ij}(\theta)$	Fisher information matrix / Riemannian metric
$\tilde{\nabla}$	Natural gradient
$A(\theta)$	Log-partition function of exponential family
$D_{\mathrm{KL}}$	Kullback-Leibler divergence

Theorems

Theorem 1: Chentsov Uniqueness Theorem

The Fisher information metric is (up to a positive scalar multiple) the unique Riemannian metric on the simplex of probability distributions that is invariant under sufficient statistics (Markov morphisms). Any statistically meaningful Riemannian metric on the space of distributions must be proportional to the Fisher metric.

Theorem 2: Pythagorean Theorem in Information Geometry

For an e-flat submanifold S and a point p, the KL divergence decomposes as D_KL(p || q) = D_KL(p || p*) + D_KL(p* || q) for all q in S, where p* is the m-projection of p onto S (the closest point under m-geodesics). This generalises the Pythagorean theorem to divergence geometry.

Theorem 3: Natural Gradient and Fisher Efficiency

The natural gradient direction tilde-nabla L(theta) = G(theta)^{-1} nabla L(theta) is the direction of steepest ascent in the Riemannian manifold with Fisher metric. Natural gradient descent on the log-likelihood achieves Fisher efficiency: the updates have minimal variance among all unbiased gradient-based estimators.

Worked Examples

1
Log-density: log p(x; mu, sigma^2) = -1/2 * log(2*pi*sigma^2) - (x-mu)^2/(2*sigma^2).
2
Score function components: d/d mu log p = (x-mu)/sigma^2, d/d sigma^2 log p = -(1/(2*sigma^2)) + (x-mu)^2/(2*sigma^4).
$\frac{\partial \log p}{\partial \mu} = \frac{x-\mu}{\sigma^2},\quad \frac{\partial \log p}{\partial \sigma^2} = -\frac{1}{2\sigma^2} + \frac{(x-\mu)^2}{2\sigma^4}$
3
Fisher matrix: g_{11} = E[(x-mu)^2/sigma^4] = 1/sigma^2, g_{22} = E[(1/(2*sigma^2) - (x-mu)^2/(2*sigma^4))^2] = 1/(2*sigma^4), g_{12} = 0.
$G = \begin{pmatrix} 1/\sigma^2 & 0 \\ 0 & 1/(2\sigma^4) \end{pmatrix}$

✓ Answer

The Fisher matrix for N(mu, sigma^2) is diagonal: G = diag(1/sigma^2, 1/(2*sigma^4)). The geometry of the Gaussian family is the Poincare upper half-plane.

Practice Problems

Hardfree response

Explain the duality between exponential families and mixture families in information geometry.

Common Mistakes

Common Mistake

Thinking KL divergence is a Riemannian distance.

KL divergence is asymmetric (D_KL(p||q) != D_KL(q||p)) and does not satisfy the triangle inequality. It is a divergence (contrast function), not a metric. However, its second-order Taylor expansion around p gives the Fisher metric: D_KL(p||p+dp) ≈ (1/2) g_ij dp_i dp_j.

Quiz

The Fisher information matrix in information geometry is:

A measure of the entropy of the distributionThe Riemannian metric on the statistical manifoldThe Hessian of the log-likelihoodThe covariance matrix of the parameters

Historical Background

The geometric approach to statistics traces to Rao (1945), who introduced the Fisher information metric. Efron (1975) connected this to statistical curvature and higher-order inference. Amari systematised the field in his 1985 book 'Differential-Geometrical Methods in Statistics', introducing the alpha-geometry, dual connections, and the information-geometric structure of exponential families. Chentsov (1982) proved that the Fisher metric is the unique Riemannian metric on the space of distributions invariant under sufficient statistics.

1945
Rao introduces the Fisher-Rao Riemannian metric on statistical models
C.R. Rao
1975
Efron develops the theory of statistical curvature
Bradley Efron
1985
Amari's monograph establishes information geometry systematically
Shun-ichi Amari
1982
Chentsov proves uniqueness of the Fisher metric among invariant metrics
Nikolai Chentsov

Summary

Information geometry treats families of probability distributions as Riemannian manifolds with the Fisher-Rao metric.
The natural gradient = G(theta)^{-1} nabla L moves in the statistically natural direction on the manifold.
Exponential families are e-flat; mixture families are m-flat; their duality underlies the EM algorithm.
Chentsov's theorem: the Fisher metric is the unique invariant Riemannian metric on distributions.

References

BookAmari, S. Differential-Geometrical Methods in Statistics. Springer, 1985.
BookAmari, S. and Nagaoka, H. Methods of Information Geometry. AMS, 2000.
WebsiteWikipedia -- Information geometry