Stochastic Control Theory — Mathematics Knowledge System

Overview

Stochastic control theory studies optimal decision-making for systems driven by stochastic processes, particularly stochastic differential equations. The goal is to find a control policy u(t, x) that minimizes an expected cost functional over a continuous-time system. The Hamilton-Jacobi-Bellman (HJB) equation characterizes the value function, generalizing the Bellman equation to continuous time. Stochastic control underlies optimal portfolio selection (Merton problem), LQG (linear-quadratic-Gaussian) control, and stochastic optimal control in engineering.

Intuition

Think of driving a car on a slippery road: the wheel controls the direction, but the road noise introduces randomness. You want to get to your destination with minimum fuel and risk. At each moment, you choose a control (steering angle) that trades off immediate cost (fuel) against future expected cost (remaining distance + risk). The HJB equation encodes this tradeoff: the optimal control at each (t, x) balances drift toward the goal against the random perturbations.

Formal Definition

Definition

Consider a controlled SDE dX_t = f(t, X_t, u_t) dt + sigma(t, X_t, u_t) dB_t. The value function V(t, x) = inf_u E[int_t^T c(s, X_s, u_s) ds + g(X_T) | X_t = x] satisfies the Hamilton-Jacobi-Bellman PDE.

dX_t = f(t, X_t, u_t)\,dt + \sigma(t, X_t, u_t)\,dB_t \quad \text{(controlled SDE)}

Controlled state equation

V(t,x) = \inf_{u} \mathbb{E}\!\left[\int_t^T c(s,X_s,u_s)\,ds + g(X_T)\,\Big|\, X_t = x\right]

Value function

-\frac{\partial V}{\partial t} = \inf_{u}\left[c(t,x,u) + f(t,x,u)\frac{\partial V}{\partial x} + \frac{\sigma^2(t,x,u)}{2}\frac{\partial^2 V}{\partial x^2}\right]

Hamilton-Jacobi-Bellman (HJB) equation

V(T, x) = g(x) \quad \text{(terminal condition)}

Terminal condition

u^*(t,x) = \operatorname*{arg\,min}_{u}\left[c(t,x,u) + f(t,x,u) V_x + \frac{\sigma^2}{2} V_{xx}\right]

Optimal control (feedback form)

Notation

Notation	Meaning
$V(t,x)$	Value function: minimum expected cost from state x at time t
$u^*(t,x)$	Optimal control in feedback form
$H(t,x,u,p,M)$	Hamiltonian: c(t,x,u) + f(t,x,u)p + (sigma^2/2)M where p = V_x, M = V_{xx}

Theorems

Theorem 1: Theorem 1 (HJB Equation -- Verification Theorem)

\text{Suppose } V \in C^{1,2}([0,T] \times \mathbb{R}) \text{ satisfies the HJB PDE with terminal condition } V(T,x) = g(x). \text{ Then } V(t,x) \le J(t,x;u) \text{ for all admissible } u. \text{ If the minimizer } u^*(t,x) \text{ is admissible, then } V = J(\cdot; u^*), \text{ and } u^* \text{ is optimal.}

Theorem 2: Theorem 2 (Merton's Optimal Portfolio)

\text{For a CRRA investor with utility } U(w) = w^\gamma / \gamma, \;\gamma < 1, \gamma \ne 0, \text{ investing in a risky asset (GBM) and cash, the optimal fraction of wealth in the risky asset is constant: } \pi^* = \frac{\mu - r}{(1-\gamma)\sigma^2}, \text{ independent of wealth and time.}

Worked Examples

1
Terminal condition: V(T, x) = x^2/2. Cost rate: c(t,x,u) = u^2/2. Dynamics: f(t,x,u) = u, sigma constant.
$\text{HJB: } -V_t = \min_u \left[\frac{u^2}{2} + u V_x\right] + \frac{\sigma^2}{2}V_{xx}$
2
Minimize over u: take derivative of u^2/2 + u V_x with respect to u, set to zero: u + V_x = 0, so u* = -V_x.
$u^*(t,x) = -V_x(t,x)$
3
Substituting u* back: -V_t = -(V_x)^2/2 + sigma^2 V_{xx}/2. Guess V(t,x) = a(t) x^2/2 + b(t). Then V_x = a(t) x, V_{xx} = a(t), V_t = a'(t) x^2/2 + b'(t). The u* = -a(t)x.
$-a'(t)\frac{x^2}{2} - b'(t) = -\frac{a(t)^2 x^2}{2} + \frac{\sigma^2 a(t)}{2}$
4
Matching coefficients: -a'(t) = -a(t)^2, so a'(t) = a(t)^2, with a(T) = 1. This ODE: a(t) = 1/(T-t+1). Optimal control: u*(t,x) = -x/(T-t+1).
$a(t) = \frac{1}{T - t + 1}, \quad u^*(t,x) = -\frac{x}{T - t + 1}$

✓ Answer

Optimal control u*(t,x) = -x/(T-t+1). The control is proportional to the state with a time-varying gain that increases as T approaches.

Practice Problems

Hardfree response

Write the HJB equation for the problem: dX_t = u_t dt + dB_t, minimize E[int_0^T (X_t^2 + u_t^2) dt + X_T^2]. Identify the optimal control structure.

Hardfree response

Explain the difference between open-loop and closed-loop (feedback) optimal controls in stochastic control.

Common Mistakes

Common Mistake

Treating the HJB equation like an ordinary ODE

The HJB is a nonlinear PDE. It often has no classical smooth solution (viscosity solutions are needed). For LQG problems, the quadratic guess works; generally, numerical methods are required.

Common Mistake

Forgetting the Ito correction (1/2) sigma^2 V_{xx} in the HJB

The generator of the controlled diffusion includes the second-order term (sigma^2/2) V_{xx}. Omitting it gives the deterministic Pontryagin/HJB, not the stochastic version.

Quiz

Summary

Stochastic control: minimize E[int cost dt + terminal cost] subject to a controlled SDE.
HJB equation: -V_t = min_u [c + f V_x + (sigma^2/2) V_{xx}], V(T,x) = g(x).
Optimal control u*(t,x) is the arg min of the Hamiltonian -- a state-feedback (Markov) policy.
LQG: quadratic costs + linear dynamics + Gaussian noise => Riccati equation for gain.
Merton problem: constant Merton ratio pi* = (mu-r)/((1-gamma)*sigma^2) for power utility.

References

BookFleming, W. H. and Rishel, R. W. -- Deterministic and Stochastic Optimal Control
BookMerton, R. C. -- Continuous-Time Finance, Chapter 5
WebsiteWikipedia -- Stochastic control