Probability

The birthday problem asks: in a group of n randomly chosen people, what is the probability that at least two share the same birthday (assuming 365 equally likely birthdays and ignoring leap years)? Counterintuitively, only 23 people are needed for this probability to exceed 50%, far fewer than most people's intuition suggests. The trick is that the number of PAIRS of people grows quadratically with n (there are C(n,2) pairs), so the chance that at least one pair matches becomes surprisingly large even for modest group sizes. It is easiest to compute via the complement: the probability that ALL birthdays are different, then subtract from 1.

A continuous probability distribution describes a random variable that can take any value in an interval of real numbers, such as height, time, or temperature. Unlike discrete distributions, the probability of any single exact value is 0; instead, probabilities are computed as areas under a probability density function (PDF) f(x) over intervals. A valid PDF must be nonnegative everywhere and integrate to 1 over its entire support. Common continuous distributions include the uniform, normal (Gaussian), exponential, and gamma distributions, each arising from different physical or statistical modeling assumptions.

A discrete probability distribution describes the probabilities of a random variable that can take only a countable number of values (finite or countably infinite), such as the outcomes of a die roll or the number of customer arrivals in an hour. It is characterized by a probability mass function (PMF) p(x) = P(X = x), which assigns a probability to each possible value, with all probabilities nonnegative and summing to 1. Common examples include the Bernoulli, binomial, Poisson, and geometric distributions, each modeling a different kind of discrete random process. The cumulative distribution function F(x) = P(X ≤ x) accumulates these probabilities and is a step function that jumps at each possible value of X.

The geometric distribution models the number of independent Bernoulli(p) trials needed to get the first success. In the common 'number of trials' formulation, X takes values 1, 2, 3, ... with PMF P(X=k) = (1-p)^{k-1}p, since the first k-1 trials must fail and the k-th must succeed. It is the discrete analog of the exponential distribution and is the unique discrete distribution with the memoryless property: given that the first success hasn't occurred by trial n, the additional number of trials needed still follows the same geometric distribution. Its mean is 1/p and its variance is (1-p)/p².

The hypergeometric distribution models the number of successes in a fixed number of draws from a finite population WITHOUT replacement, where the population contains a known number of successes and failures. If a population of size N contains K successes, and n items are drawn without replacement, the number of successes X follows a hypergeometric distribution with PMF P(X=k) = C(K,k)C(N-K,n-k)/C(N,n). This contrasts with the binomial distribution, which assumes draws are independent (with replacement); the hypergeometric distribution accounts for the fact that each draw changes the composition of the remaining population.

The continuous uniform distribution on an interval [a,b] assigns equal probability density to every point in the interval, meaning any subinterval of a given length is equally likely regardless of where it sits within [a,b]. Its probability density function is f(x) = 1/(b-a) for a ≤ x ≤ b and 0 elsewhere, reflecting 'no value is more likely than any other' within the range. The mean is the midpoint (a+b)/2, and the variance is (b-a)²/12. The uniform distribution is fundamental in simulation, since most random number generators produce values that approximate Uniform(0,1), which can then be transformed into samples from other distributions.

Covariance measures how two random variables change together: Cov(X,Y) = E[(X-μ_X)(Y-μ_Y)] = E[XY] - E[X]E[Y]. A positive covariance means X and Y tend to be above (or below) their means together; a negative covariance means one tends to be above its mean when the other is below. Because covariance's magnitude depends on the units and scale of X and Y, it is often normalized into the correlation coefficient ρ = Cov(X,Y)/(σ_Xσ_Y), which is unitless and always lies between -1 and 1. Independent random variables always have zero covariance, but zero covariance does not imply independence — it only rules out linear association.

A joint probability distribution describes the probability behavior of two or more random variables simultaneously. For discrete random variables X and Y, the joint PMF p(x,y) = P(X=x, Y=y) gives the probability of every pair of outcomes, and it must sum to 1 over all pairs. From the joint distribution, the marginal distribution of X alone is recovered by summing over all values of Y: p_X(x) = Σ_y p(x,y). Two random variables are independent exactly when their joint PMF factors as the product of their marginals, p(x,y) = p_X(x)p_Y(y) for all x, y.

Two events A and B are independent if the occurrence of one does not change the probability of the other: P(A|B) = P(A), or equivalently P(A ∩ B) = P(A)P(B). This multiplicative rule is the defining test for independence, and it extends to collections of more than two events, where mutual independence requires the product rule to hold for every subcollection, not just pairwise. Independence is a modeling assumption, not something you can verify from a single outcome — it says that learning about B gives no information about A. Independent events are distinct from mutually exclusive events; in fact, two events with positive probability cannot be both mutually exclusive and independent.

The law of total probability expresses the probability of an event A in terms of conditional probabilities given a partition of the sample space. If B₁, B₂, ..., Bₙ partition the sample space (they are mutually exclusive and their union is the entire space, each with positive probability), then P(A) = Σᵢ P(A|Bᵢ)P(Bᵢ). This lets you compute an overall probability by breaking a problem into cases, finding the probability of A within each case, and weighting by how likely each case is. It is one of the most widely used tools in probability for problems involving multiple scenarios, stages, or sources.

In probability theory, the sample space of an experiment or random trial is the set of all possible outcomes or results of that experiment. A sample space is usually denoted using set notation, and the possible ordered outcomes, or sample points, are listed as elements in the set. It is common to refer to a sample space by the labels S, Ω, or U. The elements of a sample space may be numbers, words, letters, or symbols, and can be finite, countably infinite, or uncountably infinite. An event is any subset of the sample space — a collection of outcomes we're interested in assigning a probability to.

In probability theory, conditional probability is a measure of the probability of an event occurring, given that another event (by assumption, presumption, assertion, or evidence) is already known to have occurred. This particular method relies on event A occurring with some sort of relationship with another event B. If the event of interest is A and the event B is known or assumed to have occurred, 'the conditional probability of A given B', or 'the probability of A under the condition B', is written as P(A|B). This can also be understood as the fraction of probability of B that intersects with A — the ratio of the probability of both events happening to the probability of the given event happening.

A random variable is a mathematical formalization of a quantity or object which depends on random events. The term 'random variable' in its mathematical definition refers to neither randomness nor variability but instead is a function whose domain is the set of possible outcomes in a sample space and whose range is typically a subset of the real numbers. A random variable lets us attach a number to every outcome of an experiment, so we can apply arithmetic, calculus, and statistics to what would otherwise just be abstract outcomes.

In probability theory, the expected value (also called the expectation, mean, or first moment) is a generalization of the weighted average. It summarizes a random variable's distribution with a single number: the long-run average value you'd observe if you repeated the underlying random experiment infinitely many times. Unlike a simple average, the expected value weights each possible outcome by its probability, not by equal weight.

The law of large numbers (LLN) states that as the number of independent, identically distributed trials of a random experiment grows, the sample average of the results converges to the expected value of the underlying distribution. There are two versions: the weak law, which guarantees convergence in probability, and the strong law, which guarantees almost-sure convergence. The LLN is the theoretical justification for why casinos profit reliably over many bets despite the randomness of any single bet, and why simulation and sampling methods (like Monte Carlo estimation) produce accurate results with enough trials.

The moment generating function (MGF) of a random variable X is M(t) = E[e^{tX}], defined for all t where this expectation is finite. It is called a 'moment generating' function because its derivatives at t=0 recover the moments of X: the k-th derivative evaluated at 0 equals E[X^k]. MGFs are useful because they uniquely determine a distribution (when they exist in a neighborhood of 0), and the MGF of a sum of independent random variables is the product of their individual MGFs, which makes MGFs a powerful tool for finding the distribution of sums.

The variance of a random variable X measures how spread out its values are around its expected value. It is defined as the expected value of the squared deviation from the mean, Var(X) = E[(X − μ)²], where μ = E[X]. An equivalent and often more convenient computational form is Var(X) = E[X²] − (E[X])². Variance is always nonnegative, and it is zero exactly when X is constant (almost surely). Its square root, the standard deviation σ = √Var(X), expresses spread in the same units as X itself, which is why it is often reported alongside the mean.

Monte Carlo methods are a broad class of computational algorithms that use repeated random sampling to estimate numerical quantities that are difficult or impossible to compute exactly, such as complex integrals, expectations, or probabilities. The core idea is to simulate a random process many times and use the sample average of the outcomes as an estimate of the true quantity, relying on the law of large numbers to guarantee that this average converges to the correct value as the number of simulations grows. The method takes its name from the Monte Carlo Casino, evoking the role of chance, and was formalized by Stanislaw Ulam and John von Neumann in the 1940s for nuclear physics calculations.

Bayes' theorem describes how to update the probability of a hypothesis given new evidence. It relates the conditional probability P(A|B) to the reverse conditional P(B|A), letting you flip the direction of conditioning — essential whenever you observe an effect and want to infer a cause.