Singular Value Decomposition — Mathematics Knowledge System

Overview

The singular value decomposition (SVD) factors ANY m×n matrix A — even non-square, non-invertible ones — as A = UΣVᵀ, where U and V are orthogonal matrices and Σ is diagonal with nonnegative entries called singular values. Unlike eigendecomposition, which only applies (nicely) to square diagonalizable matrices, SVD exists for every real matrix, which is why it is the single most versatile matrix factorization in numerical linear algebra. The singular values measure how much A stretches space along each of a special set of orthogonal input/output direction pairs, and they underlie low-rank approximation, PCA, and pseudo-inverses.

Intuition

SVD says that every linear map, however it distorts space, can be broken into three simple stages: rotate/reflect (Vᵀ), stretch along the coordinate axes by nonnegative factors (Σ), then rotate/reflect again (U). Geometrically, this means the image of a unit sphere under any matrix A is always an ellipsoid, and the singular values are exactly the lengths of that ellipsoid's semi-axes. The largest singular value tells you the maximum possible stretching factor A can apply to any unit vector — the most 'important' direction of the transformation — while smaller singular values capture progressively less significant directions, which is exactly why truncating small singular values gives a good low-rank approximation.

Formal Definition

Definition

For any real m×n matrix A, the SVD writes:

A = U\Sigma V^\top

Singular value decomposition

U^\top U = I_m, \quad V^\top V = I_n, \quad \Sigma = \operatorname{diag}(\sigma_1,\ldots,\sigma_r,0,\ldots)

U, V orthogonal; Σ diagonal, σ_1 \ge \cdots \ge \sigma_r > 0

A^\top A\, v_i = \sigma_i^2 v_i, \qquad A v_i = \sigma_i u_i

v_i are eigenvectors of AᵀA; σ_i² are its eigenvalues

Worked Examples

Compute AᵀA.
$A^\top A = \begin{pmatrix}3&0\\0&-2\end{pmatrix}\begin{pmatrix}3&0\\0&-2\end{pmatrix} = \begin{pmatrix}9&0\\0&4\end{pmatrix}$
AᵀA is diagonal, so its eigenvalues are its diagonal entries: 9 and 4.
$\lambda_1 = 9,\ \lambda_2 = 4$
Singular values are square roots of the eigenvalues of AᵀA, sorted descending.
$\sigma_1 = \sqrt{9}=3, \quad \sigma_2 = \sqrt{4}=2$

Answer: Singular values: σ_1 = 3, σ_2 = 2.

Practice Problems

Difficulty 6/10

Find the singular values of A = [[5,0],[0,0]].

Difficulty 7/10

Given A = [[2,0],[0,3]], find its SVD (U, Σ, V).

Difficulty 7/10

A recommendation system compresses a large user-item ratings matrix using the top k singular values (truncated SVD). Why does keeping only the largest singular values give a good low-rank approximation, and what does discarding the small ones represent?

Quiz

Summary

SVD factors any real matrix as A = UΣVᵀ with U, V orthogonal and Σ diagonal with nonnegative singular values.
Singular values are square roots of the eigenvalues of AᵀA, measuring stretching along special orthogonal axis pairs.
Truncated SVD gives the provably best low-rank approximation of A, underlying PCA, compression, and recommendation systems.

References

WebsiteWikipedia — Singular value decomposition