Floating-Point Error — Mathematics Knowledge System

Overview

Floating-point error is the discrepancy between an exact real-number computation and what a computer actually produces, because computers represent real numbers with only a finite number of significant binary digits (the IEEE 754 double-precision format uses about 15–17 significant decimal digits). Every arithmetic operation can introduce a small rounding error, and these errors accumulate and sometimes amplify through a long computation. The most dangerous failure mode is catastrophic cancellation: subtracting two nearly equal floating-point numbers can wipe out most of the significant digits, leaving a result dominated by rounding noise. Understanding floating-point error is essential for writing numerically stable algorithms — ones where small input or rounding errors do not blow up into large output errors.

Intuition

A computer's floating-point numbers are like a ruler with only so many tick marks — very fine near the values it's built to measure, but still finite. Ordinary arithmetic on such approximate numbers accumulates tiny rounding errors, usually harmless on their own. But subtracting two numbers that are almost equal is like trying to measure the thickness of a coin by weighing a truck with and without it: the truck's rounding error swamps the tiny quantity you actually wanted, because the significant digits that matched between the two numbers cancel out and what's left is dominated by noise from the rounding. This is why numerically stable algorithms are often rewritten to avoid subtracting nearly-equal quantities.

Formal Definition

Definition

IEEE 754 double precision represents a nonzero real number with a relative rounding error bounded by the unit roundoff (machine epsilon) u:

\text{fl}(x) = x(1+\delta), \quad |\delta| \le u \approx 2.22\times 10^{-16}

Representable value with relative rounding error bounded by machine epsilon

\text{fl}(a \ominus b) = (a-b)(1+\delta)

Rounding error in a single floating-point subtraction

\text{relative error of } a-b \approx \dfrac{u \cdot \max(|a|,|b|)}{|a-b|}

Cancellation amplification when a ≈ b

Worked Examples

Discriminant: b²−4ac = 100000² − 4(1)(1) = 10,000,000,000 − 4 = 9,999,999,996.
$b^2-4ac = 9{,}999{,}999{,}996$
√discriminant ≈ 99999.99998, very close to b = 100000 — subtracting these nearly-equal numbers loses precision.
$\sqrt{b^2-4ac} \approx 99999.99998$
Naive small root: (−b+√disc)/(2a) = (−100000+99999.99998)/2.
$x_{\text{naive}} = \dfrac{-100000+99999.99998}{2} \approx -1.0000003 \times 10^{-5}$
Stable alternative avoids the cancellation by using x₁ = 2c/(−b−√disc), which involves adding two same-signed numbers instead of subtracting nearly-equal ones.
$x_{\text{stable}} = \dfrac{2c}{-b-\sqrt{b^2-4ac}} \approx -1.0000000001 \times 10^{-5}$

Answer: The naive formula gives ≈ −1.0000003×10⁻⁵ while the stable formula gives ≈ −1.0000000001×10⁻⁵ — the naive result has lost several significant digits to catastrophic cancellation, even though both start from the same discriminant.

Practice Problems

Difficulty 3/10

Explain, without computing, why (1 − cos(x))/x² computed directly in floating-point becomes wildly inaccurate for very small x (e.g. x = 10⁻⁸), even though the true limit as x→0 is 1/2.

Difficulty 3/10

Catastrophic cancellation occurs when:

Multiplying two large numbersSubtracting two nearly equal floating-point numbersAdding many small numbers of the same signDividing by a large number

Difficulty 5/10

A structural engineer computes a beam's small deflection as the difference of two large, nearly equal displacement values (each around 10⁶ mm) measured to double precision. Why should they be cautious about the result, and what's a common fix?

Quiz

Summary

Floating-point numbers have finite precision (about 15–16 significant decimal digits in IEEE 754 double), so every arithmetic operation can introduce small rounding error.
Catastrophic cancellation — subtracting two nearly equal numbers — can destroy most significant digits and dominate the result with rounding noise.
Numerically stable algorithms are deliberately reformulated to avoid subtracting nearly-equal quantities, keeping rounding error under control.

References

WebsiteWikipedia — Floating-point arithmetic