Mathematics.

multivariate probability

Covariance and Correlation

Probability30 minDifficulty4 out of 10

You should know: variance of a random variable, joint probability distributions

Overview

Covariance measures how two random variables change together: Cov(X,Y) = E[(X-μ_X)(Y-μ_Y)] = E[XY] - E[X]E[Y]. A positive covariance means X and Y tend to be above (or below) their means together; a negative covariance means one tends to be above its mean when the other is below. Because covariance's magnitude depends on the units and scale of X and Y, it is often normalized into the correlation coefficient ρ = Cov(X,Y)/(σ_Xσ_Y), which is unitless and always lies between -1 and 1. Independent random variables always have zero covariance, but zero covariance does not imply independence — it only rules out linear association.

Intuition

Imagine plotting many (X,Y) pairs on a scatterplot. If the cloud of points trends upward — high X tends to pair with high Y — the covariance is positive. If it trends downward, covariance is negative. If there's no discernible linear trend, covariance is near zero. Covariance itself doesn't tell you HOW STRONG the trend is, because its size depends on the scale of X and Y (measuring height in millimeters instead of meters would inflate the covariance). Correlation fixes this by dividing out the standard deviations, squashing the measure into the fixed range [-1, 1] so +1 means a perfect increasing line, -1 a perfect decreasing line, and 0 means no linear relationship.

Formal Definition

Definition

For random variables X and Y with means μ_X, μ_Y and standard deviations σ_X, σ_Y:

Cov⁡(X,Y)=E[(X−μX)(Y−μY)]=E[XY]−E[X]E[Y]\operatorname{Cov}(X,Y) = E[(X-\mu_X)(Y-\mu_Y)] = E[XY] - E[X]E[Y]
Covariance
ρX,Y=Cov⁡(X,Y)σXσY,−1≤ρX,Y≤1\rho_{X,Y} = \frac{\operatorname{Cov}(X,Y)}{\sigma_X \sigma_Y}, \qquad -1 \le \rho_{X,Y} \le 1
Correlation coefficient
Var⁡(X+Y)=Var⁡(X)+Var⁡(Y)+2Cov⁡(X,Y)\operatorname{Var}(X+Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X,Y)
Variance of a sum

Worked Examples

  1. Compute the means of X and Y.

    E[X]=1+2+33=2,E[Y]=2+4+63=4E[X] = \tfrac{1+2+3}{3} = 2, \quad E[Y] = \tfrac{2+4+6}{3} = 4

  2. Compute E[XY].

    E[XY]=1⋅2+2⋅4+3⋅63=2+8+183=283E[XY] = \tfrac{1\cdot2 + 2\cdot4 + 3\cdot6}{3} = \tfrac{2+8+18}{3} = \tfrac{28}{3}

  3. Apply the covariance shortcut formula.

    Cov⁡(X,Y)=283−2⋅4=283−8=43\operatorname{Cov}(X,Y) = \tfrac{28}{3} - 2\cdot4 = \tfrac{28}{3} - 8 = \tfrac{4}{3}

Answer: Cov(X,Y) = 4/3 ≈ 1.33 (positive, since Y = 2X exactly).

Practice Problems

Difficulty 4/10

Var(X) = 16, Var(Y) = 25, Cov(X,Y) = -10. Find ρ_{X,Y}.

Difficulty 4/10

Var(X) = 9, Var(Y) = 4, Cov(X,Y) = 2. Find Var(X - Y).

Difficulty 5/10

A portfolio has two stocks with Var(X)=0.04, Var(Y)=0.09, and correlation ρ=0.6. Find Cov(X,Y) and Var(X+Y).

Quiz

The correlation coefficient ρ is always in the range:
If X and Y are independent, then Cov(X,Y) is:
Var(X+Y) equals:

Summary

  • Cov(X,Y) = E[XY] - E[X]E[Y] measures the direction of the linear relationship between X and Y.
  • Correlation ρ = Cov(X,Y)/(σ_Xσ_Y) normalizes covariance to always fall in [-1,1].
  • Zero covariance follows from independence, but zero covariance alone does not imply independence.

References

  1. WebsiteWikipedia — Covariance and correlation