Least-Squares Approximation — Mathematics Knowledge System

Overview

Least-squares approximation finds the curve from a chosen family (most often a straight line or low-degree polynomial) that best fits a set of data points when no curve can pass through all of them exactly — which is the usual situation with noisy measurements. 'Best' means minimizing the sum of the squared vertical distances (residuals) between the data and the fitted curve, a criterion introduced by Legendre (1805) and Gauss (who used it in 1795 and later gave its probabilistic justification). For a straight-line fit this is ordinary linear regression; more generally, fitting any linear-in-its-coefficients model to overdetermined data reduces to solving a system of linear equations called the normal equations. Least squares is ubiquitous — used to fit trend lines to economic data, calibrate sensors, reconcile surveying measurements, and initialize orbit determination in astronomy.

Intuition

Imagine a scatterplot of noisy data with no single line running through every point. Any candidate line misses some points high and some low; measure each miss (the residual) and square it so positive and negative misses don't cancel and big misses count extra. Least squares picks the one line that makes the total of all those squared misses as small as possible. Squaring rather than, say, taking absolute values isn't arbitrary — it makes the minimization problem solvable by calculus (setting derivatives to zero gives a clean linear system, the normal equations) and it penalizes a few very bad misses more than many small ones, matching the usual assumption that errors are small and roughly symmetric.

Formal Definition

Definition

Given data points (x₁,y₁), …, (xₙ,yₙ) and a model y = mx + b, least squares chooses (m, b) to minimize the sum of squared residuals S; setting the partial derivatives of S to zero yields the normal equations:

S(m,b) = \sum_{i=1}^{n}\big(y_i - (mx_i+b)\big)^2

Sum of squared residuals (objective to minimize)

m = \dfrac{n\sum x_iy_i - \sum x_i \sum y_i}{n \sum x_i^2 - \left(\sum x_i\right)^2}, \qquad b = \dfrac{\sum y_i - m\sum x_i}{n}

Closed-form slope and intercept

A^{\mathsf T}A\,\hat{x} = A^{\mathsf T} b

Normal equations for the general linear model Ax ≈ b

Worked Examples

Compute the needed sums over n = 4 points.
$\sum x_i = 10,\quad \sum y_i = 8,\quad \sum x_i^2 = 30,\quad \sum x_iy_i = 23$
Apply the slope formula.
$m = \dfrac{4(23) - (10)(8)}{4(30) - 10^2} = \dfrac{92-80}{120-100} = \dfrac{12}{20} = 0.6$
Apply the intercept formula.
$b = \dfrac{8 - 0.6(10)}{4} = \dfrac{8-6}{4} = \dfrac{2}{4} = 0.5$

Answer: y = 0.6x + 0.5.

Practice Problems

Difficulty 4/10

Given Σxᵢ=10, Σyᵢ=8, Σxᵢ²=30, Σxᵢyᵢ=23, n=4 (as above), verify that plugging m=0.6, b=0.5 into the normal equations Σyᵢ = mΣxᵢ + nb and Σxᵢyᵢ = mΣxᵢ² + bΣxᵢ is consistent.

Difficulty 5/10

Fit a least-squares line to the points (0,1), (1,3), (2,4).

Difficulty 6/10

A lab measures a spring's extension (mm) under loads (N): (1,1), (2,2), (3,2), (4,3) — the same data fitted above, y = 0.6x + 0.5. Using Hooke's law intuition, interpret the slope and intercept physically.

Quiz

Summary

Least squares fits a model (e.g., y = mx + b) to overdetermined data by minimizing the sum of squared residuals S = Σ(yᵢ − ŷᵢ)².
Setting ∂S/∂m = ∂S/∂b = 0 gives the normal equations, with closed-form slope and intercept formulas for the linear case.
For (1,1),(2,2),(3,2),(4,3) the fitted line is y = 0.6x + 0.5, with residuals −0.1, 0.3, −0.3, 0.1 and minimum squared-error sum S = 0.2.
In matrix form the general normal equations are AᵀA x̂ = Aᵀb, generalizing the line-fitting formulas to any linear-in-coefficients model.

References

WebsiteWikipedia — Least squares
WebsiteWikipedia — Linear least squares