Mathematics.

regression and correlation

Multiple Regression

Statistics30 minDifficulty4 out of 10

You should know: linear regression, correlation coefficient

Overview

Multiple regression extends simple linear regression to model a response variable y as a linear function of two or more predictor variables x₁, x₂, ..., xₚ simultaneously. Each predictor gets its own coefficient, estimated (typically via least squares) to minimize the sum of squared residuals across all observations. The coefficient on each predictor represents its effect on y while holding all other predictors constant, which allows the model to separate out the individual contribution of each variable even when predictors are correlated with each other. Multiple regression is widely used to control for confounding variables, make predictions from several inputs at once, and quantify how much variance in y (measured by R²) is jointly explained by the full set of predictors.

Intuition

Simple linear regression asks 'how does y change as x changes?' but multiple regression asks the more refined question 'how does y change as xⱼ changes, if we hold all the other predictors fixed?' This matters because predictors are often correlated with each other in real data — for example, years of education and income might both be correlated with age. Multiple regression's coefficients isolate each predictor's own contribution after accounting for the others, which is why it's the standard tool for controlling for confounding variables in observational studies.

Formal Definition

Definition

For n observations with p predictors, the multiple regression model is:

yi=β0+β1xi1+β2xi2+⋯+βpxip+εiy_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip} + \varepsilon_i
Multiple regression model
β^=(XTX)−1XTy\hat{\beta} = (X^TX)^{-1}X^Ty
Ordinary least squares estimator (matrix form)
R2=1−SSresSStotR^2 = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}}
Coefficient of determination

Worked Examples

  1. Substitute the values into the fitted equation.

    y^=10+2(5)+3(4)=10+10+12\hat{y} = 10 + 2(5) + 3(4) = 10 + 10 + 12

  2. Sum the terms.

    y^=32\hat{y} = 32

Answer: ŷ = 32.

Practice Problems

Difficulty 3/10

For the model ŷ = 5 + 1.5x₁ - 2x₂, predict y when x₁ = 4 and x₂ = 3.

Difficulty 4/10

A regression yields SS_total = 200 and SS_residual = 40. Find R².

Difficulty 5/10

A researcher studying salary uses a simple regression on years of experience alone (R² = 0.30), then adds education level as a second predictor and gets R² = 0.55. What does the increase in R² tell us, and can R² ever decrease when adding a predictor in ordinary least squares?

Quiz

In multiple regression, the coefficient βⱼ on predictor xⱼ represents:
Why might a researcher prefer multiple regression over several separate simple regressions?
What happens to R² (not adjusted R²) when an additional predictor is added to an OLS regression model?

Summary

  • Multiple regression models a response variable as a linear combination of several predictors simultaneously, estimated via least squares.
  • Each coefficient represents the effect of its predictor holding all other predictors constant, which helps control for confounding.
  • R² measures the proportion of variance jointly explained by all predictors, and it never decreases when a new predictor is added in OLS.

References

  1. WebsiteWikipedia — Linear regression