Gradient — Mathematics Knowledge System

Overview

The gradient ∇f of a multivariable function f collects all its partial derivatives into a single vector. That vector has a striking geometric meaning: it points in the direction of steepest increase of f at a given point, and its magnitude tells you exactly how steep that climb is. The gradient is the multivariable generalization of the derivative and is the single most important object in multivariable optimization, from Lagrange multipliers to gradient descent in machine learning.

Intuition

Standing on a hillside z=f(x,y), the gradient at your location is an arrow pointing in the compass direction of steepest uphill climb, and its length tells you how steep that climb is. Walk in the gradient's direction and you gain elevation fastest; walk perpendicular to it and you stay at the same elevation (that's the direction of the level curve through your point). Every other direction gives some intermediate rate of ascent or descent — the gradient is the single direction that 'wins' the steepest-ascent competition.

Formal Definition

Definition

The gradient of f(x,y,z) (or however many variables) is the vector of all its first partial derivatives:

\nabla f = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z}\right)

Generalizes to n variables as an n-component vector

Gradient in ℝ³

D_{\mathbf{u}}f = \nabla f \cdot \mathbf{u}, \qquad |\mathbf{u}|=1

The rate of change of f in the direction of unit vector u

Directional derivative via the gradient

Notation

Notation	Meaning
$\nabla f$	'Nabla f' or 'grad f' — the gradient vector of f
$\operatorname{grad} f$	Alternative notation for the gradient
$\nabla = \left(\frac{\partial}{\partial x}, \frac{\partial}{\partial y}, \frac{\partial}{\partial z}\right)$	The del/nabla operator itself, a vector of partial-derivative operators

Derivation

Deriving why ∇f points in the direction of steepest ascent: the directional derivative of f in direction of unit vector u is Dᵤf = ∇f·u. Using the dot product's geometric form:

D_{\mathbf u}f = \nabla f \cdot \mathbf u = |\nabla f|\,|\mathbf u|\cos\phi = |\nabla f|\cos\phi

φ is the angle between ∇f and u; |u|=1 since it's a unit vector

D_{\mathbf u}f \text{ is maximized when } \cos\phi = 1, \text{ i.e. } \phi = 0

cos φ ≤ 1 always, with equality exactly when u points in the same direction as ∇f

\implies \max_{\mathbf u} D_{\mathbf u}f = |\nabla f|, \text{ attained when } \mathbf{u} = \frac{\nabla f}{|\nabla f|}

The maximum rate of increase is |∇f|, achieved by walking exactly in the direction of ∇f

Proofs

The gradient is orthogonal to level curves/surfaces

$\text{Let } \mathbf{r}(t) = (x(t), y(t)) \text{ be any curve lying on the level curve } f(x,y)=k$ (Parametrize a path constrained to stay at constant height k)
$f(x(t), y(t)) = k \text{ for all } t$ (Definition of the curve lying on the level set)
$\frac{d}{dt}f(x(t),y(t)) = \frac{\partial f}{\partial x}x'(t) + \frac{\partial f}{\partial y}y'(t) = \nabla f \cdot \mathbf{r}'(t) = 0$ (Chain rule (multivariable), and derivative of the constant k is 0)
$\nabla f \cdot \mathbf{r}'(t) = 0 \implies \nabla f \perp \mathbf{r}'(t)$ (A zero dot product between two nonzero vectors means they are perpendicular)
$\text{Since } \mathbf{r}'(t) \text{ is tangent to the level curve, } \nabla f \text{ is normal (perpendicular) to the level curve at every point}$ (r'(t) is by construction tangent to any curve lying in the level set, and this holds for every such curve)

Properties

Linearity

\nabla(af+bg) = a\nabla f + b\nabla g

Product rule

\nabla(fg) = f\nabla g + g\nabla f

Critical points

\nabla f(\mathbf{x}_0) = \mathbf{0} \text{ at local maxima, minima, and saddle points (necessary condition)}

Steepest descent

-\nabla f \text{ points in the direction of steepest decrease, the basis of gradient descent}

Theorems

Theorem 1: Steepest ascent theorem

\text{The direction of maximum increase of } f \text{ at a point is } \nabla f, \text{ with maximum rate of increase } |\nabla f|

Theorem 2: Gradient orthogonality to level sets

\nabla f(\mathbf{x}_0) \text{ is orthogonal to the level curve/surface of } f \text{ passing through } \mathbf{x}_0

Applications

Gradient descent updates parameters θ by stepping in the direction −∇L(θ), the direction of steepest decrease of the loss function, iteratively minimizing loss.

3D Visualization

Animation

Animates a ball rolling downhill on a surface z=f(x,y), with an arrow at each position showing −∇f (steepest descent direction) — the ball's actual path visually tracks the negative gradient direction at every point, illustrating gradient descent.

Worked Examples

Compute the partial derivatives.
$\frac{\partial f}{\partial x} = 2xy, \qquad \frac{\partial f}{\partial y} = x^2+\cos(y)$
Evaluate at (1,0).
$\nabla f(1,0) = (2\cdot1\cdot0,\ 1^2+\cos 0) = (0, 2)$

Answer: ∇f(1,0) = (0, 2)

Practice Problems

Difficulty 5/10

Find ∇f for f(x,y,z) = xyz + z².

Difficulty 7/10

A metal plate's temperature is T(x,y) = 100 − x² − 2y². Find the direction of fastest temperature increase at (1,1), and the rate.

Difficulty 6/10

Find all critical points of f(x,y) = x²+y²−4x+6y+13 by setting ∇f=0.

Common Mistakes

Common Mistake

Thinking the gradient is a scalar (just 'the derivative') rather than a vector.

∇f is a VECTOR with both magnitude (steepness) and direction (which way is steepest). Confusing it with a partial derivative loses the directional information that makes the gradient useful for optimization.

Common Mistake

Assuming −∇f always leads all the way to the global minimum in one step.

−∇f only gives the LOCAL direction of steepest descent at the current point; gradient descent must take many small steps, re-evaluating the gradient each time, and can still get stuck in local minima or saddle points.

Common Mistake

Forgetting to normalize u to a unit vector when computing a directional derivative via ∇f·u.

The formula Dᵤf = ∇f·u requires |u|=1. If you use a non-unit vector, the dot product no longer equals the true directional derivative — it must be rescaled by dividing by |u|.

Quiz

The gradient ∇f at a point is always:

Tangent to the level curve through that pointPerpendicular to the level curve through that pointEqual to zero at every pointParallel to the x-axis

What is the maximum possible directional derivative of f at a point?

∇f · (1,1)|∇f|f(point)0

Flashcards

1 / 4

Historical Background

The gradient emerged from 19th-century work in vector analysis and physics. James Clerk Maxwell used gradient-like operators in his 1873 Treatise on Electricity and Magnetism to describe how scalar potentials produce vector fields (e.g., electric field as the negative gradient of electric potential). The modern ∇ ('nabla' or 'del') notation was introduced by William Rowan Hamilton in the context of quaternions and later adapted into standard vector calculus notation by Gibbs and Heaviside in the 1880s-1890s.

1837
Hamilton introduces the nabla operator symbol in the context of quaternion calculus
William Rowan Hamilton
1873
Maxwell uses gradient operators systematically in electromagnetic theory
James Clerk Maxwell
1880s-1890s
Gibbs and Heaviside develop modern vector calculus notation, including ∇f for the gradient
Josiah Willard Gibbs, Oliver Heaviside

Summary

The gradient ∇f = (∂f/∂x, ∂f/∂y, ...) collects all partial derivatives into one vector.
∇f points in the direction of steepest ascent of f, and |∇f| is the maximum rate of increase.
∇f is always perpendicular to the level curve/surface of f passing through that point.
Directional derivative: Dᵤf = ∇f · u for any unit vector u.
Foundation of gradient descent (machine learning), conservative force fields (physics), and Lagrange multipliers (constrained optimization).

References

BookStewart, J. Calculus: Early Transcendentals, 8th ed. Ch. 14.
BookMaxwell, J.C. (1873). A Treatise on Electricity and Magnetism.
WebsiteWikipedia — Gradient