Error Propagation and Numerical Stability — Mathematics Knowledge System

Overview

Every numerical computation carries two distinct kinds of error: input error already present in the data (measurement uncertainty, rounding from an earlier step) and the way each subsequent arithmetic operation propagates and possibly amplifies that error. A stable algorithm keeps errors from growing much faster than the problem's own inherent sensitivity (its condition number) would demand; an unstable algorithm can turn tiny, unavoidable input errors into huge output errors even when the underlying mathematical problem is perfectly well-conditioned. The most common culprit is catastrophic cancellation: subtracting two nearly equal floating-point numbers destroys most of their significant digits, since the small true difference is represented with the same absolute (not relative) precision as the large original numbers. Understanding error propagation is what separates a numerically responsible implementation of a formula from one that quietly returns garbage — the quadratic formula, variance calculations, and finite-difference derivatives are classic examples where the 'obvious' formula is numerically unstable and a reformulated version is preferred in practice.

Intuition

Think of each arithmetic operation as passing a measurement through a noisy amplifier. Multiplying and dividing are relatively gentle amplifiers — the relative (percentage) errors of the inputs simply add, so a 1% error stays roughly a 1% error. Subtraction of two nearly equal numbers, however, is a wildly unstable amplifier: if two numbers agree to 6 significant digits and you subtract them, the result keeps only the last, uncertain digits — like weighing two nearly identical heavy suitcases on a scale accurate to the nearest gram, then trying to learn the tiny weight difference between the contents of each from that single measurement. The absolute error stayed the same size, but because the answer itself shrank enormously, the relative error exploded. Numerically stable algorithms are designed to dodge this trap — by algebraically rearranging a formula (e.g., rationalizing rather than directly subtracting nearly-equal square roots) so no step ever needs to subtract two close numbers.

Formal Definition

Definition

For a differentiable function f of inputs subject to small absolute errors, first-order (linearized) error propagation bounds the resulting absolute and relative error in the output:

\delta f \approx \left|\dfrac{\partial f}{\partial x}\right|\delta x + \left|\dfrac{\partial f}{\partial y}\right|\delta y \quad \text{(first-order absolute error propagation)}

General propagation formula for f(x,y)

f(x,y) = xy \;\Rightarrow\; \delta f \approx |y|\,\delta x + |x|\,\delta y, \qquad \dfrac{\delta f}{|f|} \approx \dfrac{\delta x}{|x|} + \dfrac{\delta y}{|y|}

Multiplication: relative errors add

\text{fl}(a - b) = (a-b)(1+\varepsilon), \quad |\varepsilon| \le u \;\Rightarrow\; \text{rel. error} \approx u\cdot\dfrac{|a|+|b|}{|a-b|}

Catastrophic cancellation: relative error is amplified when a ≈ b

Worked Examples

Apply the product error-propagation formula δf ≈ |y|δx + |x|δy.
$\delta f \approx |3.00|(0.01) + |2.00|(0.02) = 0.03 + 0.04 = 0.07$
Compute f itself and then the relative error.
$f = xy = 6.00, \qquad \dfrac{\delta f}{f} = \dfrac{0.07}{6.00} \approx 0.01167 = 1.167\%$
Cross-check using the relative-error addition rule directly: relative errors of x and y are 0.01/2=0.5% and 0.02/3≈0.667%.
$0.5\% + 0.667\% = 1.167\% \;\checkmark$

Answer: δf ≈ 0.07, so f = 6.00 ± 0.07, a relative error of about 1.167% — matching the sum of the input relative errors, confirming multiplication is a well-behaved (non-amplifying) operation for error.

Practice Problems

Difficulty 5/10

Given x = 5.0 ± 0.1 and y = 4.0 ± 0.2, estimate the absolute error in f = xy using δf ≈ |y|δx + |x|δy.

Difficulty 6/10

For the same x=5.0±0.1, y=4.0±0.2, what is the relative error in f = xy, computed by adding the input relative errors?

Difficulty 7/10

The 'naive' quadratic formula x = (−b + √(b²−4ac))/(2a) suffers catastrophic cancellation when b² ≫ 4ac and b > 0 (so −b+√(b²−4ac) subtracts two nearly equal large numbers). Explain the standard numerically stable fix.

Quiz

Summary

Numerical error propagation tracks how input uncertainty (measurement or rounding error) grows through a sequence of arithmetic operations.
Multiplication/division are well-behaved: relative errors approximately add, e.g. x=2.00±0.01 times y=3.00±0.02 gives f=6.00±0.07 (about 1.167% relative error).
Subtracting nearly equal numbers causes catastrophic cancellation: e.g. 1.234567 − 1.234561 = 0.000006 keeps the same absolute uncertainty but the relative error jumps from ~0.00004% to ~8.3%.
A numerically stable algorithm avoids amplifying error beyond what the problem's own condition number requires; the standard fix for the cancellation-prone quadratic formula uses Vieta's product-of-roots identity to compute the second root safely.

References

WebsiteWikipedia — Numerical stability
WebsiteWikipedia — Catastrophic cancellation
BookBurden, R. L. & Faires, J. D. Numerical Analysis, 10th ed. Cengage, 2015.