Mathematics.

multivariate probability

Joint Probability Distributions

Probability30 minDifficulty4 out of 10

You should know: random variables, independence of events

Overview

A joint probability distribution describes the probability behavior of two or more random variables simultaneously. For discrete random variables X and Y, the joint PMF p(x,y) = P(X=x, Y=y) gives the probability of every pair of outcomes, and it must sum to 1 over all pairs. From the joint distribution, the marginal distribution of X alone is recovered by summing over all values of Y: p_X(x) = Σ_y p(x,y). Two random variables are independent exactly when their joint PMF factors as the product of their marginals, p(x,y) = p_X(x)p_Y(y) for all x, y.

Intuition

Instead of a single probability table for one random variable, imagine a full grid: rows for every possible value of X, columns for every possible value of Y, and each cell holding the probability of that exact (X,Y) combination. Summing along a row or column collapses the grid down to a single variable's distribution — that's the marginal. If the grid's cell values are exactly what you'd get by multiplying a row-total by a column-total, the two variables carry no information about each other, which is precisely independence.

Formal Definition

Definition

For discrete random variables X and Y with joint PMF p(x,y):

p(x,y)=P(X=x,Y=y),∑x∑yp(x,y)=1p(x,y) = P(X=x, Y=y), \qquad \sum_x \sum_y p(x,y) = 1
Joint PMF and normalization
pX(x)=∑yp(x,y)p_X(x) = \sum_y p(x,y)
Marginal distribution of X
p(x,y)=pX(x) pY(y)(if X, Y independent)p(x,y) = p_X(x)\,p_Y(y) \quad \text{(if X, Y independent)}
Independence factorization
E[XY]=∑x∑yxy p(x,y)E[XY] = \sum_x\sum_y xy\,p(x,y)
Expectation of the product

Worked Examples

  1. Each of the 4 equally likely outcomes (HH,HT,TH,TT) gives a joint probability of 1/4.

    p(1,1)=p(1,0)=p(0,1)=p(0,0)=14p(1,1)=p(1,0)=p(0,1)=p(0,0) = \tfrac{1}{4}

  2. Sum over Y to get the marginal of X: p_X(1) = p(1,1)+p(1,0).

    pX(1)=14+14=12p_X(1) = \tfrac{1}{4}+\tfrac{1}{4} = \tfrac{1}{2}

Answer: The joint PMF is uniform (1/4 each), and the marginal P(X=1) = 1/2, matching a fair coin.

Practice Problems

Difficulty 4/10

Given joint PMF p(0,0)=0.1, p(0,1)=0.2, p(1,0)=0.3, p(1,1)=0.4, find the marginal P(X=1).

Difficulty 5/10

Using the same table (p(0,0)=0.1, p(0,1)=0.2, p(1,0)=0.3, p(1,1)=0.4), check whether X and Y are independent.

Difficulty 5/10

A store tracks whether customers buy coffee (X: 1=yes,0=no) and a pastry (Y: 1=yes,0=no). Joint probabilities are p(0,0)=0.4, p(0,1)=0.1, p(1,0)=0.2, p(1,1)=0.3. Find the marginal probability a customer buys coffee.

Quiz

The marginal distribution of X is obtained from the joint PMF by:
X and Y are independent if and only if:
A joint PMF p(x,y) over all pairs (x,y) must satisfy:

Summary

  • A joint PMF p(x,y) = P(X=x,Y=y) gives probabilities over pairs of outcomes and sums to 1.
  • Marginals are recovered by summing the joint PMF over the other variable.
  • X and Y are independent exactly when the joint PMF factors as p_X(x)·p_Y(y) for every pair.

References

  1. WebsiteWikipedia — Joint probability distribution