Mathematics.

distributions

Mixture Distributions

Probability35 minDifficulty6 out of 10

You should know: continuous probability distributions, discrete probability distributions

Overview

A mixture distribution arises when a random variable's distribution is a weighted combination of two or more component distributions. Intuitively, one first randomly selects a component according to mixing weights, then draws an observation from the selected component. Mixture models are ubiquitous in data science (Gaussian mixture models for clustering), actuarial science (modelling heterogeneous populations), and Bayesian statistics (prior-predictive distributions). They can capture multi-modality, heavy tails, and other complex shapes that a single parametric family cannot.

Intuition

Suppose a factory has two machines: machine A produces bolts with diameter ~ N(10, 0.1²) and machine B produces bolts with diameter ~ N(10.5, 0.2²). If 70% of bolts come from A and 30% from B, the diameter of a randomly selected bolt follows a mixture: 0.7·N(10, 0.01) + 0.3·N(10.5, 0.04). The resulting distribution will be bimodal — two bumps — reflecting the two machines. A mixture is the distribution you see when you sample from a heterogeneous population without knowing which subpopulation each observation belongs to.

Formal Definition

Definition

A random variable X follows a mixture distribution with K components if its density (or PMF) can be written as:

fX(x)=∑k=1Kπk fk(x),πk≥0,∑k=1Kπk=1f_X(x) = \sum_{k=1}^{K} \pi_k\, f_k(x), \quad \pi_k \geq 0,\quad \sum_{k=1}^K \pi_k = 1
Mixture density
E[X]=∑k=1Kπk μk,μk=E[X∣Z=k]E[X] = \sum_{k=1}^K \pi_k\, \mu_k, \quad \mu_k = E[X \mid Z=k]
Mean of a mixture
Var⁡(X)=∑k=1Kπkσk2+∑k=1Kπk(μk−μ)2\operatorname{Var}(X) = \sum_{k=1}^K \pi_k \sigma_k^2 + \sum_{k=1}^K \pi_k (\mu_k - \mu)^2
Variance of a mixture (within-component + between-component)

Worked Examples

  1. 1

    Mixture mean: weighted average of component means.

    E[X]=0.6(0)+0.4(5)=2E[X] = 0.6(0) + 0.4(5) = 2
  2. 2

    Component variances: σ₁² = 1, σ₂² = 1. Within-component variance:

    ∑πkσk2=0.6(1)+0.4(1)=1\sum \pi_k \sigma_k^2 = 0.6(1) + 0.4(1) = 1
  3. 3

    Between-component variance (spread of means around μ = 2):

    ∑πk(μk−μ)2=0.6(0−2)2+0.4(5−2)2=0.6(4)+0.4(9)=2.4+3.6=6\sum \pi_k (\mu_k - \mu)^2 = 0.6(0-2)^2 + 0.4(5-2)^2 = 0.6(4) + 0.4(9) = 2.4 + 3.6 = 6
  4. 4

    Total variance = within + between.

    Var⁡(X)=1+6=7\operatorname{Var}(X) = 1 + 6 = 7

✓ Answer

E[X] = 2, Var(X) = 7.

Practice Problems

Mediumfree response

A mixture of two exponentials: with probability 0.3 draw from Exp(1) (mean 1) and with probability 0.7 draw from Exp(0.5) (mean 2). Find the mean of the mixture.

Mediumfree response

A Gaussian mixture has weights π₁ = 0.5, π₂ = 0.5 and component means μ₁ = −2, μ₂ = 2 with component variances σ₁² = σ₂² = 1. Find the overall mean and variance.

Quiz

The density of a K-component mixture is:
The variance of a mixture distribution equals:
Mixture models can capture which feature that single parametric distributions often cannot?

Summary

  • A mixture distribution has density f(x) = Σπₖfₖ(x) where πₖ ≥ 0 and Σπₖ = 1.
  • E[X] = Σπₖμₖ (weighted average of component means).
  • Var(X) = within-component variance + between-component variance (law of total variance).
  • Mixture models capture heterogeneous populations, multi-modality, and heavy tails.

References

  1. WebsiteWikipedia — Mixture distribution