Mathematics.

multivariate distributions

Order Statistics

Probability45 minDifficulty7 out of 10

You should know: continuous probability distributions, joint probability distributions

Overview

Order statistics arise when the observations in a random sample are arranged from smallest to largest. The k-th order statistic X_{(k)} of a sample of size n is the k-th smallest value. Order statistics appear throughout probability and statistics: the sample minimum X_{(1)} and maximum X_{(n)} are the extreme order statistics; the sample median is the middle order statistic; they underlie non-parametric methods, reliability theory, and auction theory. The distribution of any order statistic can be derived from the underlying continuous CDF.

Intuition

To understand the density of X_{(k)}, think combinatorially: for the k-th order statistic to equal x, exactly k−1 of the remaining n−1 observations must fall below x, exactly n−k must fall above x, and one observation lands at x. The multinomial coefficient n!/[(k−1)!(n−k)!1!] counts the ways to assign the n observations to these three categories, [F(x)]^{k−1} is the probability that k−1 fall below, [1−F(x)]^{n−k} is the probability that n−k fall above, and f(x)dx is the probability of one landing at x.

Formal Definition

Definition

Let X₁,...,Xₙ be i.i.d. continuous random variables with CDF F and density f. The k-th order statistic X_{(k)} (the k-th smallest of X₁,...,Xₙ) has density:

fX(k)(x)=n!(k−1)!(n−k)! [F(x)]k−1[1−F(x)]n−kf(x)f_{X_{(k)}}(x) = \frac{n!}{(k-1)!(n-k)!}\,[F(x)]^{k-1}[1-F(x)]^{n-k}f(x)
Density of the k-th order statistic
fX(1)(x)=n[1−F(x)]n−1f(x)f_{X_{(1)}}(x) = n[1-F(x)]^{n-1}f(x)
Density of the minimum (k=1)
fX(n)(x)=n[F(x)]n−1f(x)f_{X_{(n)}}(x) = n[F(x)]^{n-1}f(x)
Density of the maximum (k=n)
FX(k)(x)=∑j=kn(nj)[F(x)]j[1−F(x)]n−jF_{X_{(k)}}(x) = \sum_{j=k}^{n} \binom{n}{j} [F(x)]^j [1-F(x)]^{n-j}
CDF of the k-th order statistic

Worked Examples

  1. 1

    For Uniform(0,1): F(x) = x and f(x) = 1 for 0 < x < 1. Use the formula for k = n = 5.

    fX(5)(x)=5[F(x)]5−1f(x)=5x4⋅1=5x4,0<x<1f_{X_{(5)}}(x) = 5[F(x)]^{5-1}f(x) = 5x^4 \cdot 1 = 5x^4, \quad 0 < x < 1
  2. 2

    Verify this integrates to 1: ∫₀¹ 5x⁴ dx = [x⁵]₀¹ = 1. ✓

✓ Answer

f_{X_{(5)}}(x) = 5x⁴ for 0 < x < 1.

Practice Problems

Mediumfree response

X₁, X₂, X₃ are i.i.d. Uniform(0,1). Find the density of the minimum X_{(1)}.

Mediumfree response

For i.i.d. Uniform(0,1) with n = 4, find E[X_{(2)}] (the second-smallest).

Quiz

The density of the maximum X_{(n)} of n i.i.d. variables with CDF F and density f is:
For i.i.d. Uniform(0,1) with sample size n, E[X_{(k)}] equals:
The k-th order statistic of a Uniform(0,1) sample follows which distribution?

Summary

  • X_{(k)} is the k-th smallest of n i.i.d. observations; its density is n!/[(k−1)!(n−k)!]·F(x)^{k−1}·[1−F(x)]^{n−k}·f(x).
  • Special cases: minimum (k=1) has density n[1−F]^{n−1}f; maximum (k=n) has density n·F^{n−1}f.
  • For Uniform(0,1): X_{(k)} ~ Beta(k, n−k+1) with E[X_{(k)}] = k/(n+1).

References

  1. WebsiteWikipedia — Order statistic