Mathematics.

limit theorems

Law of Large Numbers

Probability30 minDifficulty4 out of 10

You should know: expectation, variance of a random variable

Overview

The law of large numbers (LLN) states that as the number of independent, identically distributed trials of a random experiment grows, the sample average of the results converges to the expected value of the underlying distribution. There are two versions: the weak law, which guarantees convergence in probability, and the strong law, which guarantees almost-sure convergence. The LLN is the theoretical justification for why casinos profit reliably over many bets despite the randomness of any single bet, and why simulation and sampling methods (like Monte Carlo estimation) produce accurate results with enough trials.

Intuition

Flip a fair coin once and you might get heads or tails — no useful pattern. Flip it a million times, and the fraction of heads will almost certainly be extremely close to 50%, even though any individual flip is still a coin toss. The 'noise' from individual trials doesn't disappear, but it gets averaged out and shrinks relative to the signal as more trials pile up — the variance of the sample mean is σ²/n, which shrinks toward 0 as n grows, forcing the sample average to settle near the true mean.

Formal Definition

Definition

Let X₁, X₂, ... be independent, identically distributed random variables with E[Xᵢ] = μ, and let the sample mean be:

Xˉn=1n∑i=1nXi\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n} X_i
Sample mean
Xˉn→Pμas n→∞\bar{X}_n \xrightarrow{P} \mu \quad \text{as } n \to \infty
Weak law of large numbers (convergence in probability)
P ⁣(lim⁡n→∞Xˉn=μ)=1P\!\left(\lim_{n\to\infty}\bar{X}_n = \mu\right) = 1
Strong law of large numbers (almost sure convergence)
Var⁡(Xˉn)=σ2n\operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n}
Variance of the sample mean shrinks with n

Worked Examples

  1. By the LLN, the sample average converges toward the true mean 3.5 as n increases.

    Xˉn→3.5\bar{X}_n \to 3.5

  2. Compute the standard deviation of the sample mean using Var(X̄ₙ) = σ²/n.

    Var⁡(Xˉ10000)=35/1210000=35120000≈0.000292\operatorname{Var}(\bar{X}_{10000}) = \frac{35/12}{10000} = \frac{35}{120000} \approx 0.000292

  3. Take the square root to get the standard deviation.

    σXˉ≈0.000292≈0.0171\sigma_{\bar{X}} \approx \sqrt{0.000292} \approx 0.0171

Answer: The sample average approaches 3.5, with spread shrinking to about ±0.017 at n=10,000.

Practice Problems

Difficulty 4/10

A random variable has mean 10 and variance 16. Find the variance of the sample mean of 100 independent observations.

Difficulty 4/10

A random variable has mean 10 and variance 16. What is the standard deviation of the sample mean of 400 independent observations?

Difficulty 5/10

A casino game has expected profit $2 per play for the house. Explain, using the law of large numbers, why the casino's average profit per play becomes very predictable over thousands of plays even though each individual play is random.

Quiz

The law of large numbers describes the behavior of:
As sample size n increases, the variance of the sample mean X̄ₙ:
The 'strong' law of large numbers differs from the 'weak' law in that it guarantees:

Summary

  • As n grows, the sample mean of i.i.d. trials converges to the true expected value μ.
  • The variance of the sample mean shrinks as σ²/n, which is the mechanism behind the convergence.
  • The LLN justifies why casinos, insurers, and simulations (Monte Carlo methods) get reliable long-run averages from randomness.

References

  1. WebsiteWikipedia — Law of large numbers