de Finetti's Theorem — Mathematics Knowledge System

Overview

De Finetti's theorem is one of the most profound results in probability: it characterizes infinite exchangeability as equivalent to being a mixture of i.i.d. processes. Specifically, if X_1, X_2, ... is infinitely exchangeable, then there exists a random probability measure mu (the de Finetti measure) such that, conditioning on mu, the X_i are i.i.d. with distribution mu. This theorem provides the mathematical foundation for Bayesian inference: priors are de Finetti mixing measures and data are conditionally i.i.d. given the parameter.

Intuition

De Finetti's theorem says: if you cannot tell the order, there must be hidden information. If two flips look symmetric (exchangeable), there is a 'hidden coin bias' theta such that once you know theta, the flips ARE independent. The prior distribution over theta is exactly the de Finetti mixing measure. This turns Bayesian reasoning (prior + likelihood) into a mathematical theorem rather than a philosophical choice.

Formal Definition

Definition

Let X_1, X_2, ... be an infinitely exchangeable sequence of {0,1}-valued random variables. Then there exists a unique probability measure Q on [0,1] (the de Finetti measure) such that for all n and all (x_1,...,x_n) in {0,1}^n: P(X_1=x_1,...,X_n=x_n) = integral_0^1 theta^{sum x_i} (1-theta)^{n - sum x_i} Q(d theta).

P(X_1=x_1,\ldots,X_n=x_n) = \int_0^1 \theta^{\sum_{i=1}^n x_i}(1-\theta)^{n-\sum_{i=1}^n x_i}\,Q(d\theta)

De Finetti representation (binary)

\theta = \lim_{n\to\infty} \frac{X_1 + \cdots + X_n}{n} \quad \text{a.s.} \quad (\text{the empirical frequency})

Theta as limiting frequency

P(X_1=x_1,\ldots,X_n=x_n) = \int \prod_{i=1}^n p(x_i\mid\theta)\,\mu(d\theta)

General de Finetti representation

X_1,X_2,\ldots \text{ infinitely exchangeable} \iff X_i \mid \theta \overset{\text{i.i.d.}}{\sim} P_\theta \text{ for some random } \theta

De Finetti's theorem

Notation

Notation	Meaning
$Q$	De Finetti mixing measure: the distribution of the latent parameter theta
$\theta$	Latent parameter (e.g., coin bias): the a.s. limit of empirical frequency
$\mu(d\theta)$	Prior distribution in Bayesian statistics = de Finetti measure

Theorems

Theorem 1: Theorem 1 (De Finetti, Binary Case)

\text{A sequence } \{X_n\} \text{ of } \{0,1\}\text{-valued random variables is infinitely exchangeable if and only if there exists a probability measure } Q \text{ on } [0,1] \text{ such that } P(X_1=x_1,\ldots,X_n=x_n) = \int_0^1 \theta^{s_n}(1-\theta)^{n-s_n}\,Q(d\theta), \text{ where } s_n = \sum x_i. \text{ Moreover, } Q \text{ is the law of } \lim_{n\to\infty} (X_1+\cdots+X_n)/n.

Theorem 2: Theorem 2 (General De Finetti -- Hewitt-Savage)

\text{Let } X_1, X_2, \ldots \text{ be infinitely exchangeable random variables taking values in a Polish space. Then there exists a random probability measure } \tilde{\mu} \text{ on the state space such that, conditionally on } \tilde{\mu}, \text{ the } X_i \text{ are i.i.d. with distribution } \tilde{\mu}.

Worked Examples

1
By the de Finetti representation, P(X_1 = 1) = E_Q[theta] = integral_0^1 theta Q(d theta).
$P(X_1 = 1) = \mathbb{E}_Q[\theta] = \int_0^1 \theta\,Q(d\theta)$
2
For Q = Beta(alpha, beta), E[theta] = alpha/(alpha+beta).
$P(X_1 = 1) = \frac{\alpha}{\alpha + \beta}$

✓ Answer

P(X_1 = 1) = alpha/(alpha + beta), the mean of the Beta prior.

Practice Problems

Mediumfree response

Explain how de Finetti's theorem justifies the Bayesian approach to statistics: 'assume data are i.i.d. given an unknown parameter theta, with a prior on theta.'

Hardfree response

Describe the de Finetti measure Q for the case where Q is a point mass at theta_0. What does the sequence X_1, X_2, ... look like?

Common Mistakes

Common Mistake

Thinking de Finetti's theorem requires the X_i to actually be i.i.d.

De Finetti's theorem says the X_i are conditionally i.i.d. GIVEN theta. Marginally, they are exchangeable but correlated (Cov(X_i, X_j) = Var(theta) > 0 if Q is non-degenerate).

Common Mistake

Applying de Finetti to finitely exchangeable sequences

De Finetti's representation theorem requires INFINITE exchangeability. Finitely exchangeable sequences (like sampling without replacement) do NOT have a de Finetti representation in general.

Quiz

Summary

De Finetti's theorem: X_1, X_2, ... is infinitely exchangeable iff X_i | theta are i.i.d. for some random theta.
The de Finetti measure Q is the law of theta = lim (X_1+...+X_n)/n (a.s. limit).
Bayesian prior = de Finetti measure: exchangeability assumption uniquely determines the prior.
Beta-Binomial: Beta prior is conjugate to i.i.d. Bernoulli; posterior is Beta(alpha+k, beta+n-k).
Infinite vs finite exchangeability: only infinite exchangeability gives the de Finetti representation.

References

Bookde Finetti, B. -- Theory of Probability, Vol. 1 (1974)
BookAldous, D. -- Exchangeability and Related Topics (Lecture Notes in Mathematics 1117)
WebsiteWikipedia -- De Finetti's theorem