Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method, and is effectively synonymous with the linear regression model.

Description

A linear model is expressed as either (univariate) or (multivariate with k terms). Either way, a crucial assumption is that the expected value of the error term is 0, such that the first moment is E[y_i|x_i] = α + βx_i.

Single Regression

In the case of a single predictor, the OLS regression is:

This formulation leaves the components explicit: the y-intercept term is the mean outcome at x=0, and the slope term is marginal change to the outcome per a unit change in x.

The derivation can be seen here.

Multiple Regression

In the case of multiple predictors, the regression is fit like:

But conventionally, this OLS system is solved using linear algebra as:

Note that using a b here is intentional.

The derivation can be seen here.

Estimated Coefficients

The Gauss-Markov theorem demonstrates that (with some assumptions) the OLS estimations are the best linear unbiased estimators (BLUE) for the regression coefficients. The assumptions are:

1. Linearity
2. Exogeneity, i.e. predictors are independent of the outcome and the error term
3. Random sampling

4. No perfect multicolinearity

5. Homoskedasticity, i.e. error terms are constant across observations

#5 mostly comes into the estimation of standard errors, and there are alternative estimators that are robust to heteroskedasticity.

CategoryRicottone