Stationary Processes — Mathematics Knowledge System

Overview

A stationary stochastic process is one whose statistical properties do not change over time. Strict stationarity requires all finite-dimensional distributions to be time-shift invariant; wide-sense (weak) stationarity requires only a constant mean and a covariance function that depends only on the lag. Stationary processes are fundamental to time series analysis, signal processing, and ergodic theory. The Wiener-Khinchin theorem connects the autocorrelation function to the spectral density via the Fourier transform, providing a frequency-domain analysis tool.

Intuition

A stationary process looks the same statistically at any time: its mean does not trend, its variance does not grow, and the correlation between values at times s and t depends only on their distance |t-s|, not on the absolute times. Think of a machine in steady state: measuring it today or tomorrow yields the same statistical picture. The spectral density decomposes the variance of the process by frequency, just as a prism decomposes white light into a spectrum. High spectral density at a frequency means the process oscillates strongly at that frequency.

Formal Definition

Definition

A process {X_t} is strictly stationary if all finite-dimensional distributions are shift-invariant. Wide-sense (weak) stationarity requires weaker conditions.

(X_{t_1},\ldots,X_{t_n}) \overset{d}{=} (X_{t_1+h},\ldots,X_{t_n+h}) \quad \forall\, h, t_1,\ldots,t_n

Strict stationarity

\mathbb{E}[X_t] = \mu \text{ (constant)},\quad \operatorname{Cov}(X_t, X_{t+h}) = R(h)

Wide-sense stationarity

R(h) = \mathbb{E}[(X_t - \mu)(X_{t+h} - \mu)]

Autocorrelation function (ACF)

S(f) = \int_{-\infty}^{\infty} R(h)\,e^{-2\pi i f h}\,dh

Spectral density (Fourier transform of ACF)

R(h) = \int_{-\infty}^{\infty} S(f)\,e^{2\pi i f h}\,df

Wiener-Khinchin theorem (inverse)

Notation

Notation	Meaning
$R(h)$	Autocorrelation (autocovariance) function at lag h
$S(f)$	Spectral density (power spectral density) at frequency f
$\rho(h)$	Normalized autocorrelation: rho(h) = R(h)/R(0)

Theorems

Theorem 1: Wiener-Khinchin Theorem

\text{For a wide-sense stationary process } \{X_t\} \text{ with autocovariance } R(h), \text{ the spectral density } S(f) \text{ satisfies:} \quad S(f) = \mathcal{F}\{R\}(f) = \int_{-\infty}^{\infty} R(h) e^{-2\pi i f h}\,dh \ge 0.

Theorem 2: Ergodic Theorem for Stationary Processes

\text{For an ergodic stationary process } \{X_t\}{:} \quad \frac{1}{T}\int_0^T X_t\,dt \to \mathbb{E}[X_0] \quad \text{a.s. and in } L^2 \text{ as } T \to \infty.

Theorem 3: Bochner's Theorem

\text{A continuous function } R{:}\mathbb{R} \to \mathbb{R} \text{ is the autocovariance function of a wide-sense stationary process if and only if it is positive semi-definite, i.e., } \sum_{i,j} c_i c_j R(t_i-t_j) \ge 0 \text{ for all } c_i,t_i.

Worked Examples

1
E[X_t] = 0 (since E[A] = E[B] = 0). Compute the autocovariance.
$R(h) = \mathbb{E}[X_t X_{t+h}] = \sigma^2\cos(2\pi f_0 h)$
2
The autocovariance depends only on h, so the process is wide-sense stationary. Its spectral density is the Fourier transform of R(h).
$S(f) = \frac{\sigma^2}{2}[\delta(f-f_0) + \delta(f+f_0)]$

✓ Answer

The process is wide-sense stationary with ACF R(h) = sigma^2 cos(2 pi f_0 h) and spectral density consisting of two Dirac deltas at frequencies +/-f_0.

Practice Problems

Mediumfree response

Is Brownian motion W_t wide-sense stationary? Justify your answer.

Mediumapplication

The spectral density of a wide-sense stationary process is S(f) = 1/(1 + (2 pi f)^2). Find the corresponding autocovariance R(h).

Common Mistakes

Common Mistake

Wide-sense stationarity implies strict stationarity

Wide-sense stationarity is weaker; it only constrains the first two moments. Strict stationarity constrains all finite-dimensional distributions. For Gaussian processes, they are equivalent.

Common Mistake

The spectral density S(f) must integrate to 1

The spectral density integrates to R(0) = Var(X_t), the total variance of the process, not to 1.

Quiz

Historical Background

The concept of stationarity was developed in the context of time series analysis and signal processing in the early 20th century. Norbert Wiener and Aleksandr Khinchin independently proved the theorem relating autocorrelation to spectral density in the 1930s. The ergodic theorem, proved by Birkhoff in 1931, showed that time averages equal ensemble averages for ergodic stationary processes. These results laid the foundation for modern signal processing, spectral analysis, and statistical time series methods.

1931
Birkhoff proves the ergodic theorem for stationary processes
George Birkhoff
1934
Khinchin proves the autocorrelation-spectral density theorem
Aleksandr Khinchin
1930s
Wiener develops the theory of stationary processes and spectral analysis
Norbert Wiener

Summary

A wide-sense stationary process has constant mean and autocovariance R(h) = Cov(X_t, X_{t+h}) depending only on lag h.
The Wiener-Khinchin theorem: S(f) = Fourier transform of R(h), decomposing process variance by frequency.
The ergodic theorem: for ergodic processes, time averages converge to ensemble averages.
Bochner's theorem characterizes valid autocovariance functions as positive semi-definite functions.

References

BookDurrett, R. -- Probability: Theory and Examples, 4th ed.
BookPapoulis, A. -- Probability, Random Variables, and Stochastic Processes
WebsiteWikipedia -- Stationary process