Bayes' Theorem — Mathematics Knowledge System

Overview

Bayes' theorem describes how to update the probability of a hypothesis given new evidence. It relates the conditional probability P(A|B) to the reverse conditional P(B|A), letting you flip the direction of conditioning — essential whenever you observe an effect and want to infer a cause.

Intuition

Suppose a rare disease affects 1 in 1000 people, and a test is 99% accurate. If you test positive, what's the chance you actually have the disease? Intuition says 'high', but Bayes' theorem shows it's often surprisingly low — because false positives from the huge healthy population can outnumber true positives from the tiny sick population. Bayes' theorem forces you to weigh new evidence against how likely the hypothesis was BEFORE you saw that evidence (the prior), not just how well the evidence fits the hypothesis.

Formal Definition

Definition

For events A and B with P(B) > 0:

P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}

Posterior = (Likelihood × Prior) / Evidence

Bayes' Theorem

Notation

Notation	Meaning
$P(A \mid B)$	Posterior — probability of A given that B has occurred
$P(A)$	Prior — probability of A before observing B
$P(B \mid A)$	Likelihood — probability of observing B if A is true

Derivation

Derived directly from the definition of conditional probability, applied in both directions:

P(A \cap B) = P(A \mid B)P(B) = P(B \mid A)P(A)

Both expressions equal P(A and B)

P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}

Divide both sides by P(B)

Properties

Law of total probability (expanding the denominator)

P(B) = P(B\mid A)P(A) + P(B\mid \lnot A)P(\lnot A)

Applications

Naive Bayes classifiers, spam filters, and many probabilistic machine learning models are direct applications of this theorem.

Formula Explorer

Worked Examples

P(disease)=0.01, P(positive|disease)=0.95, P(positive|healthy)=0.05.
$P(\text{disease})=0.01$
Compute P(positive) via the law of total probability.
$P(\text{pos}) = 0.95(0.01) + 0.05(0.99) = 0.059$
Apply Bayes' theorem.
$P(\text{disease}\mid \text{pos}) = \frac{0.95 \times 0.01}{0.059} \approx 0.161$

Answer: About 16.1% — much lower than the test's 95% accuracy might suggest.

Practice Problems

Difficulty 5/10

1 in 200 emails is spam. A spam filter flags 98% of spam and 2% of legitimate email. If an email is flagged, what's P(spam | flagged)?

Difficulty 6/10

A disease affects 1% of a population. A test is 99% sensitive (detects the disease) and 95% specific (correct on healthy people). If you test positive, what is the probability you actually have the disease?

Difficulty 6/10

In non-destructive testing, 2% of welds are defective. An ultrasonic test detects 90% of real defects but also flags 5% of good welds. A weld is flagged — what is the probability it is truly defective?

Common Mistakes

Common Mistake

Confusing P(A|B) with P(B|A) -- 'base rate fallacy'.

These are generally very different numbers. A 95%-accurate test does NOT mean a positive result implies 95% chance of disease — the prior probability of disease matters enormously, especially for rare conditions.

Quiz

In Bayes' theorem, what does the 'prior' P(A) represent?

The probability of A after observing evidenceThe probability of the evidence itselfThe probability of A before observing any new evidenceThe accuracy of the test

Flashcards

1 / 2

Historical Background

The theorem is named for Reverend Thomas Bayes, whose work 'An Essay towards solving a Problem in the Doctrine of Chances' was published posthumously in 1763, edited by Richard Price. The modern general form was developed independently and more thoroughly by Pierre-Simon Laplace in 1774, who applied it to problems in astronomy and demography. Bayesian statistics as a distinct methodology (as opposed to frequentist statistics) only became widely practiced in the 20th century.

1763
Bayes' essay published posthumously by Richard Price
Thomas Bayes, Richard Price
1774
Laplace independently derives and generalizes the theorem
Pierre-Simon Laplace

Summary

Bayes' theorem: P(A|B) = P(B|A)P(A) / P(B) — flips the direction of conditioning.
Posterior = Likelihood × Prior, normalized by the evidence.
The prior probability matters enormously — a highly accurate test can still yield a low posterior probability if the condition is rare (base rate fallacy).
Foundation of Bayesian statistics, spam filters, medical diagnosis, and A/B testing.

References

WebsiteWikipedia — Bayes' theorem