= Ordinary Least Squares =

'''Ordinary Least Squares''' ('''OLS''') is a linear regression method. It minimizes root mean square errors.

<<TableOfContents>>

----



== Univariate ==

Given one independent variable and one dependent (outcome) variable, the OLS model is specified as:

{{attachment:model.svg}}

It is estimated as:

{{attachment:estimate.svg}}

This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable. 

The derivation can be seen [[Statistics/OrdinaryLeastSquares/Univariate|here]].

----



== Multivariate ==

Given ''k'' independent variables, the OLS model is specified as:

{{attachment:mmodel.svg}}

It is estimated as:

{{attachment:mestimate.svg}}

More conventionally, this is estimated with [[LinearAlgebra|linear algebra]] as:

{{attachment:matrix.svg}}

The derivation can be seen [[Statistics/OrdinaryLeastSquares/Multivariate|here]].

----



== Estimated Coefficients ==

The '''Gauss-Markov theorem''' demonstrates that (with some assumptions) the OLS estimations are the '''best linear unbiased estimators''' ('''BLUE''') for the regression coefficients. The assumptions are:

 1. Linearity
 2. [[Statistics/Exogeneity|Exogeneity]]
 3. Random sampling
 4. No perfect [[LinearAlgebra/Basis|multicolinearity]]
 5. [[Statistics/Homoskedasticity|Homoskedasticity]]

The variances for each coefficient are:

{{attachment:homo1.svg}}

Note that the standard deviation of the population's parameter is unknown, so it's estimated like:

{{attachment:homo2.svg}}

If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:

{{attachment:hetero1.svg}}

Wherein, for example, ''r,,1j,,'' is the residual from regressing ''x,,1,,'' onto ''x,,2,,'', ... ''x,,k,,''.

The variances for each coefficient can be estimated with the Eicker-White formula:

{{attachment:hetero2.svg}}

See [[https://www.youtube.com/@kuminoff|Nicolai Kuminoff's]] video lectures for the derivation of the robust estimators.



----
CategoryRicottone