Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method, and is effectively synonymous with the general linear model.

Description

A simple (univariate) linear model is expressed as:

The general linear model (multivariate with k predictors) is expressed as:

Note that the predictors are treated as fixed values, not random variables. Only the outcome and error terms are random variables. Measurements of the outcome variable are also available but the random variable underlying it is modeled as a linear combination of the predictors and the error term, and those measurements are instead used to calculate residuals. Further note that, by convention, the error term is represented by a stylized epsilon ϵ (as opposed to ε).

There are crucial assumptions in this model:

• independence of all predictors

• ϵ_i ~ N(0, σ²); which is to say...

  • the error term is a normally distributed random variable

  • the variance of the error term is constant across all cases; Var[ϵ_i] = σ² for all i

  • the expected value of the error term is 0; E[ϵ_i] = 0 for all i

These enable the statement that e.g., E[y_i|x_i] = α + βx_i.

This model can be fit using the method of ordinary least squares. And furthermore this method is the best way to fit the model (under certain assumptions).

Single Regression

In the case of a single predictor, the OLS regression is:

This formulation leaves the components explicit: the y-intercept term is the mean outcome at x=0, and the slope term is marginal change to the outcome per a unit change in x.

The derivation can be seen here.

Multiple Regression

In the case of multiple predictors, the regression is fit like:

But conventionally, this OLS system is solved using linear algebra as:

Note that using a b here is intentional.

The derivation can be seen here.

Estimated Coefficients

The Gauss-Markov theorem demonstrates that (with some assumptions) the OLS estimations are the best linear unbiased estimators (BLUE) for the regression coefficients. The assumptions are:

1. Linearity
2. Exogeneity, i.e. predictors are independent of the outcome and the error term
3. Random sampling

4. No perfect multicolinearity

5. Homoskedasticity, i.e. error terms are constant across observations

#5 mostly comes into the estimation of standard errors, and there are alternative estimators that are robust to heteroskedasticity.

CategoryRicottone