Mathematics.

regression analysis

Linear Regression

Statistics30 minDifficulty5 out of 10

You should know: descriptive statistics

Overview

In statistics, linear regression is a model that estimates the relationship between a scalar response and one or more explanatory variables. A model with exactly one explanatory variable is a simple linear regression; a model with two or more explanatory variables is a multiple linear regression. This term is distinct from multivariate linear regression, which predicts multiple correlated dependent variables rather than a single dependent variable. Linear regression is one of the most widely used statistical techniques, forming the basis for prediction, trend estimation, and causal-effect modeling across nearly every quantitative field.

Intuition

Given a scatterplot of points that roughly trend upward or downward, linear regression finds the single straight line that best fits the cloud of points. 'Best' means minimizing the total squared vertical distance between each data point and the line (least squares) — this penalizes big misses much more than small ones, and has a clean closed-form solution. Once you have that line, you can predict a new response for an unseen input, and the line's slope tells you how much the response is expected to change per unit change in the predictor.

Interactive Graph

Drag m and b to fit a line to data

Loading visualization…

Formal Definition

Definition

For n observations (yᵢ, xᵢ1, ..., xᵢp), the linear regression model assumes the response is a linear combination of the predictors plus random error:

yi=β0+β1xi1+⋯+βpxip+εi,i=1,…,ny_i = \beta_0 + \beta_1 x_{i1} + \cdots + \beta_p x_{ip} + \varepsilon_i, \qquad i = 1, \ldots, n

Each response is a linear function of the predictors plus noise ε_i

Multiple linear regression model
y=Xβ+ε\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\varepsilon

Stacking all observations into vectors/matrices

Matrix form
β^=(XTX)−1XTy\hat{\boldsymbol\beta} = (\mathbf{X}^\mathsf{T}\mathbf{X})^{-1}\mathbf{X}^\mathsf{T}\mathbf{y}

Closed-form solution minimizing the sum of squared residuals

Ordinary least squares (OLS) estimator

Notation

NotationMeaning
β0\beta_0The intercept — the predicted y when all predictors are 0
β1\beta_1The slope — the expected change in y per unit increase in x₁, holding other predictors fixed
εi\varepsilon_iThe random error term for observation i, assumed to have mean 0
y^\hat{y}The fitted/predicted value from the regression line
R2R^2Coefficient of determination — the proportion of variance in y explained by the model

Properties

Least squares objective

β^=arg⁡min⁡β∑i=1n(yi−xiTβ)2\hat{\boldsymbol\beta} = \arg\min_{\boldsymbol\beta} \sum_{i=1}^n (y_i - \mathbf{x}_i^\mathsf{T}\boldsymbol\beta)^2

Simple regression slope

β^1=∑i(xi−xˉ)(yi−yˉ)∑i(xi−xˉ)2\hat{\beta}_1 = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2}

Condition: for one predictor (p=1)

Regression line passes through the means

y^=yˉ when x=xˉ\hat{y} = \bar{y} \text{ when } x = \bar{x}

Gauss–Markov theorem

Under classical assumptions (linearity, independence, homoscedasticity, zero-mean errors), OLS is the Best Linear Unbiased Estimator (BLUE)\text{Under classical assumptions (linearity, independence, homoscedasticity, zero-mean errors), OLS is the Best Linear Unbiased Estimator (BLUE)}

Applications

Econometric models regress economic outcomes (wages, GDP growth) on explanatory factors to estimate causal or associative effects.

Worked Examples

  1. Compute means: x̄ = 3, ȳ = 4.

    xˉ=3,yˉ=4\bar{x}=3, \quad \bar{y}=4

  2. Compute the slope using the covariance/variance formula.

    β^1=(1−3)(2−4)+(2−3)(3−4)+(3−3)(5−4)+(4−3)(4−4)+(5−3)(6−4)(1−3)2+(2−3)2+(3−3)2+(4−3)2+(5−3)2=4+1+0+0+44+1+0+1+4=910=0.9\hat{\beta}_1 = \frac{(1-3)(2-4)+(2-3)(3-4)+(3-3)(5-4)+(4-3)(4-4)+(5-3)(6-4)}{(1-3)^2+(2-3)^2+(3-3)^2+(4-3)^2+(5-3)^2} = \frac{4+1+0+0+4}{4+1+0+1+4} = \frac{9}{10} = 0.9

  3. Compute the intercept so the line passes through (x̄, ȳ).

    β^0=yˉ−β^1xˉ=4−0.9(3)=1.3\hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x} = 4 - 0.9(3) = 1.3

Answer: ŷ = 1.3 + 0.9x

Practice Problems

Difficulty 5/10

A regression gives ŷ = 5 + 2x. What is the predicted value when x = 10, and what does the coefficient 2 represent?

Difficulty 6/10

In ordinary least squares regression, the coefficients are chosen to minimize:

Difficulty 6/10

A lab fits concrete strength y (MPa) against curing days x and gets ŷ = 12 + 1.8x with R² = 0.94. Predict the 28-day strength and interpret R².

Common Mistakes

Common Mistake

Interpreting a strong linear regression fit (high R²) as proof of causation.

Regression captures association, not causation. A high R² only means the linear model explains a lot of the variance in y — confounding variables or reverse causation can still be at play.

Common Mistake

Extrapolating far outside the range of observed x-values and trusting the prediction.

The linear relationship is only justified within (or near) the range of the data used to fit it. Extrapolation far beyond that range can produce wildly inaccurate predictions if the true relationship isn't linear everywhere.

Quiz

Ordinary least squares (OLS) fits the line that minimizes:
A regression reports R² = 0.94. This means:
Extrapolating a fitted line far beyond the range of the data is risky because:

Summary

  • Linear regression models a response y as a linear function of predictors: y = β₀ + β₁x₁ + ... + βₚxₚ + ε.
  • Ordinary least squares (OLS) chooses coefficients that minimize the sum of squared residuals.
  • For simple regression, the slope is β̂₁ = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)², and the fitted line always passes through (x̄, ȳ).
  • R² measures the proportion of variance in y explained by the model.
  • A good fit shows association, not necessarily causation, and predictions are only reliable within the range of observed data.

References

  1. WebsiteWikipedia — Linear regression