Gaussian Processes — Mathematics Knowledge System

Overview

A Gaussian process (GP) is a collection of random variables, any finite subset of which is jointly Gaussian. A GP is completely specified by its mean function m(x) and covariance (kernel) function k(x, x'). GPs generalize the multivariate normal distribution to function spaces and provide a powerful Bayesian prior over functions. They are used in machine learning for regression (GP regression / Kriging), in geostatistics, and as a foundational object in stochastic analysis (Brownian motion is the prototype GP). Mercer's theorem characterizes the spectral properties of covariance kernels.

Intuition

A Gaussian process is a distribution over functions. Instead of specifying a prior over a single value, you specify a prior over an entire curve. The mean function m(x) gives the expected value of the function at each point; the kernel k(x, x') encodes how similar the function values at x and x' are expected to be. After observing some data, the posterior distribution over functions is also a Gaussian process (due to Gaussian conjugacy), giving exact Bayesian uncertainty quantification. The kernel choice determines the smoothness, periodicity, and other structural properties of the functions.

Formal Definition

Definition

A Gaussian process {f(x), x in X} is a stochastic process where any finite collection (f(x_1), ..., f(x_n)) is jointly Gaussian.

f \sim \mathcal{GP}(m, k) \iff (f(x_1),\ldots,f(x_n)) \sim \mathcal{N}(\mathbf{m},\mathbf{K}) \quad \forall n,\, x_1,\ldots,x_n

Gaussian process definition

m(x) = \mathbb{E}[f(x)], \quad k(x,x') = \operatorname{Cov}(f(x),f(x'))

Mean and covariance (kernel) function

k(x,x') = k(x',x) \ge 0, \quad \sum_{i,j} c_i c_j k(x_i,x_j) \ge 0

Kernel must be symmetric positive semi-definite

f \mid \mathbf{y} \sim \mathcal{GP}(m_{\text{post}}, k_{\text{post}})

Posterior GP after observing data y

m_{\text{post}}(x) = m(x) + \mathbf{k}_*^\top (\mathbf{K}+\sigma^2 I)^{-1}(\mathbf{y}-\mathbf{m})

Posterior mean (GP regression)

Notation

Notation	Meaning
$f \sim \mathcal{GP}(m,k)$	f is a Gaussian process with mean m and kernel k
$k(x,x')$	Covariance kernel function
$\mathbf{K}$	Gram matrix: K_{ij} = k(x_i, x_j)

Theorems

Theorem 1: Mercer's Theorem

\text{Let } k \text{ be a continuous positive semi-definite kernel on a compact set. Then:} \quad k(x,x') = \sum_{n=1}^{\infty} \lambda_n \phi_n(x)\phi_n(x'), \text{ where } \lambda_n \ge 0 \text{ are eigenvalues and } \phi_n \text{ are eigenfunctions of the integral operator } Tf(x) = \int k(x,x')f(x')\,dx'.

Theorem 2: GP Regression (Posterior)

\text{Given GP prior } f \sim \mathcal{GP}(0,k) \text{ and noisy observations } y_i = f(x_i) + \epsilon_i, \, \epsilon_i \sim N(0,\sigma^2), \text{ the posterior is } f \mid \mathbf{y} \sim \mathcal{GP}(m_{\text{post}}, k_{\text{post}}) \text{ with } m_{\text{post}}(x) = \mathbf{k}_*^\top(\mathbf{K}+\sigma^2 I)^{-1}\mathbf{y}.

Theorem 3: Karhunen-Loeve Expansion

\text{A GP with kernel } k \text{ on } [0,T] \text{ admits the expansion:} \quad f(x) = m(x) + \sum_{n=1}^{\infty} \sqrt{\lambda_n}\, Z_n\, \phi_n(x), \quad Z_n \overset{\text{i.i.d.}}{\sim} N(0,1), \text{ converging in } L^2.

Worked Examples

1
Any finite collection (W(t_1), ..., W(t_n)) is jointly Gaussian (sums of independent Gaussians).
$(W(t_1),\ldots,W(t_n)) \sim \mathcal{N}(\mathbf{0}, \mathbf{K})$
2
The mean function is m(t) = E[W(t)] = 0 and the kernel is Cov(W(s), W(t)) = min(s, t).
$m(t) = 0, \quad k(s,t) = \min(s,t)$

✓ Answer

Brownian motion is a GP with m(t) = 0 and kernel k(s,t) = min(s,t).

Practice Problems

Mediumfree response

Verify that k(x,x') = min(x,x') is a valid (positive semi-definite) covariance kernel for x,x' in [0,1].

Mediumapplication

Explain the role of the kernel hyperparameters in the squared exponential kernel k(x,x') = sigma_f^2 * exp(-|x-x'|^2 / (2*l^2)). What do sigma_f and l control?

Common Mistakes

Common Mistake

Any symmetric function k(x,x') is a valid kernel

A valid covariance kernel must be both symmetric and positive semi-definite: sum_ij c_i c_j k(x_i, x_j) >= 0 for all choices of c_i and x_i.

Common Mistake

GP regression gives the true function exactly at observed points

With noise variance sigma^2 > 0, the posterior mean does not interpolate; it shrinks toward the prior. Only the noise-free (sigma^2 = 0) limit gives exact interpolation.

Quiz

Historical Background

Gaussian processes as a concept in probability theory arise naturally from the multivariate Gaussian distribution and were studied by Kolmogorov, Wiener, and others in the context of prediction theory and Brownian motion in the 1930s-1940s. Krige introduced empirical versions in geostatistics (Kriging) in the 1950s. The modern machine learning perspective on GP regression was systematized by O'Hagan in the 1970s and popularized by Rasmussen and Williams in their influential 2006 book. Mercer's theorem (1909) underlies the spectral decomposition of covariance kernels.

1909
Mercer proves his theorem on positive integral operators, foundational for kernel methods
James Mercer
1951
Krige introduces Kriging (spatial interpolation via GPs) in geostatistics
Danie Krige
1978
O'Hagan formalizes GP regression for Bayesian statistics
Anthony O'Hagan
2006
Rasmussen and Williams publish the authoritative ML reference for GPs
Carl Rasmussen, Christopher Williams

Summary

A Gaussian process f ~ GP(m, k) is a distribution over functions; any finite collection is jointly Gaussian with mean m and covariance k.
Valid kernels must be symmetric and positive semi-definite; Mercer's theorem provides their spectral decomposition.
GP regression: after observing data, the posterior is a GP with analytically computable mean and kernel.
Brownian motion is the canonical GP: m(t) = 0 and k(s,t) = min(s,t).

References

BookRasmussen, C. and Williams, C. -- Gaussian Processes for Machine Learning (2006)
BookDurrett, R. -- Probability: Theory and Examples, 4th ed.
WebsiteWikipedia -- Gaussian process