Optimal Control Theory — Mathematics Knowledge System

Overview

Optimal control theory seeks a control function that drives a dynamical system from an initial state to a desired final state while minimizing (or maximizing) a performance criterion (cost functional). It generalizes the classical calculus of variations to settings where controls are constrained. The two pillars of the field are the Pontryagin Maximum Principle, which gives necessary conditions via a Hamiltonian formulation, and Bellman's Dynamic Programming Principle, which gives sufficient conditions via the Hamilton–Jacobi–Bellman equation.

Intuition

Imagine steering a rocket from launch to orbit using minimum fuel. The state is position and velocity; the control is thrust direction and magnitude. At each moment, the control must be chosen to balance current fuel burn against future cost. The Maximum Principle says: act to maximise the Hamiltonian at each instant. Dynamic Programming says: the optimal cost-to-go satisfies a PDE (HJB equation) that can be solved backward in time.

Formal Definition

Definition

The Bolza problem: minimise a cost functional combining a terminal cost and a running cost over a trajectory governed by an ODE with constrained controls.

\min_{u(\cdot)} J[u] = \phi(x(T)) + \int_0^T L(x(t), u(t), t)\, dt

Bolza cost functional

\dot{x}(t) = f(x(t), u(t), t),\quad x(0) = x_0,\quad u(t) \in U

State equation with control constraint

H(x, u, p, t) = p^T f(x, u, t) - L(x, u, t)

Pontryagin Hamiltonian

u^*(t) = \arg\max_{u \in U} H(x^*(t), u, p(t), t)

Maximum Principle (control law)

\dot{p}(t) = -\frac{\partial H}{\partial x}\bigg|_{(x^*, u^*, p)},\quad p(T) = \nabla \phi(x^*(T))

Adjoint (costate) equation

Notation

Notation	Meaning
$x(t)$	State trajectory
$u(t)$	Control function
$p(t)$	Costate (adjoint) variable
$H$	Pontryagin Hamiltonian
$V(x,t)$	Value function (optimal cost-to-go)

Theorems

Theorem 1: Pontryagin Maximum Principle

\text{If } (x^*, u^*) \text{ is optimal, there exists costate } p(t) \neq 0 \text{ such that } u^*(t) = \arg\max_{u \in U} H(x^*, u, p, t) \text{ and the costate satisfies } \dot{p} = -\partial H/\partial x

Theorem 2: Hamilton–Jacobi–Bellman Equation

-\frac{\partial V}{\partial t} = \min_{u \in U}\left[ L(x,u,t) + \nabla_x V \cdot f(x,u,t) \right],\quad V(x,T) = \phi(x)

Theorem 3: Linear Quadratic Regulator (LQR)

\text{For linear } f = Ax + Bu \text{ and quadratic cost, the optimal control is } u^*(t) = -R^{-1} B^T P(t) x(t) \text{ where } P \text{ solves the Riccati equation}

Worked Examples

1
State: (x₁, x₂) = (x, ẋ). Hamiltonian: H = p₁x₂ + p₂u − 1 (Mayer form for min time).
$H = p_1 x_2 + p_2 u - 1$
2
Maximum principle: maximise H over |u| ≤ 1, giving bang-bang control.
$u^*(t) = \operatorname{sign}(p_2(t))$
3
Costate equations: ṗ₁ = 0, ṗ₂ = −p₁, so p₂ is linear in time.
$p_2(t) = -p_1 t + c$
4
Switching once gives the classic bang-bang time-optimal solution: full thrust one direction, then the other.
$u^* = \begin{cases} +1 & t < t_s \\ -1 & t > t_s \end{cases}$

✓ Answer

The optimal control is bang-bang with a single switch — full positive thrust then full negative thrust.

Practice Problems

Hardproof writing

Derive the HJB equation from Bellman's principle of optimality.

Common Mistakes

Common Mistake

The Maximum Principle guarantees a global optimum

The PMP gives necessary conditions only. Without convexity of the Hamiltonian, there may be multiple PMP-satisfying trajectories that are only local optima.

Common Mistake

The costate variable p(t) has no physical meaning

The costate represents the shadow price of the state: p_i(t) = ∂V/∂x_i is the marginal value of relaxing the state constraint at time t.

Quiz

The Pontryagin Maximum Principle provides what type of condition for optimality?

Sufficient conditionNecessary conditionNecessary and sufficient conditionCondition valid only for linear systems

The HJB equation is solved in which direction of time?

Forward from t = 0Backward from t = TBoth simultaneouslyTime direction is irrelevant

Historical Background

Optimal control theory crystallised in the late 1950s from two independent intellectual traditions. Lev Pontryagin and his students in Moscow developed the Maximum Principle (published 1956–1962) as a Hamiltonian generalisation of the Euler–Lagrange equations for constrained controls. Simultaneously, Richard Bellman at RAND developed Dynamic Programming and the principle of optimality, published in 1957. Both approaches revolutionised aerospace, economics, and engineering control design.

1956
Pontryagin formulates the Maximum Principle for optimal control
Lev Pontryagin
1957
Bellman publishes 'Dynamic Programming', introducing the principle of optimality
Richard Bellman
1962
Pontryagin et al. publish 'The Mathematical Theory of Optimal Processes'
Lev Pontryagin, Vladimir Boltyansky, Revaz Gamkrelidze, Evgeny Mishchenko
1970s
LQR (Linear Quadratic Regulator) becomes standard in aerospace and control engineering

Summary

Optimal control minimises a functional over trajectories of a controlled ODE with constrained controls.
The Pontryagin Maximum Principle gives necessary conditions for optimality via a co-state (adjoint) variable satisfying a Hamiltonian system.
The HJB equation gives a sufficient characterisation via the value function, solved backward from the terminal condition.
LQR is the classical tractable case: linear dynamics, quadratic cost, unconstrained control — optimal feedback is linear.
Bang-bang controls arise when the Hamiltonian is linear in u and U is a convex polytope.

References

BookPontryagin, L.S., Boltyansky, V.G., Gamkrelidze, R.V. & Mishchenko, E.F. — The Mathematical Theory of Optimal Processes (1962), Wiley-Interscience
BookBellman, R. — Dynamic Programming (1957), Princeton University Press
BookFleming, W.H. & Rishel, R.W. — Deterministic and Stochastic Optimal Control (1975), Springer
WebsiteWikipedia — Optimal control
WebsiteMathWorld — Optimal Control