Mathematics.

inferential statistics

Hypothesis Testing

Statistics30 minDifficulty5 out of 10

You should know: normal distribution

Overview

A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves calculating a test statistic, then making a decision either by comparing the test statistic to a critical value or, equivalently, by evaluating a p-value computed from the test statistic. Roughly 100 specialized statistical tests are in use, covering means, proportions, variances, and more, but they all share this same underlying logic.

Intuition

Hypothesis testing is essentially a courtroom trial for a claim about the world. You start by assuming a 'null hypothesis' H₀ is true (innocent until proven guilty) — for example, 'this coin is fair' or 'this new drug has no effect'. You then ask: if H₀ really were true, how surprising would the data I actually observed be? If the data would be extremely unlikely under H₀ (quantified by a small p-value), you reject H₀ in favor of the alternative H₁. If the data isn't that surprising, you simply fail to reject H₀ — you never 'prove' H₀ true, only that you lack sufficient evidence against it.

Interactive Graph

The sampling distribution under the null hypothesis

Loading visualization…

Formal Definition

Definition

A hypothesis test specifies a null hypothesis H₀ and alternative H₁, a significance level α (the tolerance for false rejections), and a test statistic whose distribution under H₀ is known. Two types of error are possible: Type I error (rejecting a true H₀) with probability α, and Type II error (failing to reject a false H₀) with probability β.

α=P(reject H0∣H0 is true)\alpha = P(\text{reject } H_0 \mid H_0 \text{ is true})

The significance level — probability of a false positive

Type I error rate
β=P(fail to reject H0∣H1 is true)\beta = P(\text{fail to reject } H_0 \mid H_1 \text{ is true})

Probability of a false negative; 1−β is the test's power

Type II error rate
H0:p=14,H1:p>14H_0: p = \tfrac14, \qquad H_1: p > \tfrac14

Example: testing whether a proportion exceeds 1/4 (a one-sided test)

Notation

NotationMeaning
H0H_0The null hypothesis — the default claim being tested
H1H_1The alternative hypothesis
α\alphaSignificance level, typically 0.05 or 0.01 — the threshold for rejecting H₀
p-valuep\text{-value}Probability, assuming H₀ is true, of observing a test statistic at least as extreme as the one obtained

Properties

Decision rule

Reject H0 if p-value<α\text{Reject } H_0 \text{ if } p\text{-value} < \alpha

Type I / Type II tradeoff

Decreasing α (fewer false positives) generally increases β (more false negatives), for fixed sample size\text{Decreasing } \alpha \text{ (fewer false positives) generally increases } \beta \text{ (more false negatives), for fixed sample size}

Power increases with sample size

Power (1−β) generally increases as sample size n increases\text{Power } (1-\beta) \text{ generally increases as sample size } n \text{ increases}

Applications

A/B testing in software compares user engagement metrics between variants using hypothesis tests to decide which version to ship.

Worked Examples

  1. Under H₀, the probability of observing X = 25 heads out of 25 with p = 1/4 per flip is extremely small.

    P(X=25∣p=14)=(14)25≈10−15P(X = 25 \mid p = \tfrac14) = \left(\tfrac14\right)^{25} \approx 10^{-15}

  2. Since this probability is far below any reasonable α (e.g. 0.05), the observed data is essentially impossible under H₀.

    10−15≪0.0510^{-15} \ll 0.05

Answer: Reject H₀ — the data is overwhelmingly inconsistent with p = 1/4.

Practice Problems

Difficulty 4/10

A test yields a p-value of 0.03. At significance level α = 0.05, do you reject or fail to reject H₀?

Difficulty 5/10

A Type I error occurs when:

Difficulty 6/10

A supplier claims mean concrete strength ≥ 30 MPa. A sample of n = 36 cubes gives mean 28.5 MPa with σ = 4 MPa. At α = 0.05, test H₀: μ = 30 against H₁: μ < 30 using a z-test.

Common Mistakes

Common Mistake

Interpreting a p-value as 'the probability that H₀ is true'.

The p-value is P(data this extreme or more | H₀ true) — a statement about the data assuming H₀, not a probability assigned to H₀ itself. This is a very common and serious misinterpretation.

Common Mistake

Treating 'fail to reject H₀' as proof that H₀ is true.

Failing to reject only means the evidence wasn't strong enough to rule out H₀ — it does not prove H₀ is correct. Absence of evidence is not evidence of absence.

Quiz

A Type I error in hypothesis testing is:
In an acceptance test, a p-value of 0.02 with α = 0.05 means you should:
Making α smaller (say 0.05 → 0.01) to avoid false alarms tends to:

Summary

  • Hypothesis testing decides whether observed data provides sufficient evidence to reject a null hypothesis H₀ in favor of an alternative H₁.
  • The p-value is the probability, assuming H₀ is true, of seeing a result at least as extreme as the one observed.
  • Reject H₀ when the p-value is less than the chosen significance level α (commonly 0.05).
  • Type I error (false rejection, rate α) and Type II error (false non-rejection, rate β) trade off against each other.
  • Statistical significance does not mean H₀ is proven false with certainty, nor does non-significance prove H₀ true.

References

  1. WebsiteWikipedia — Statistical hypothesis test