Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method. It minimizes root mean square errors.

Univariate

The regression line passes through two points:

and

Take the generic equation form of a line:

Insert the first point into this form.

This can be trivially rewritten to solve for a in terms of b:

Insert the second point into the original form.

Now additionally insert the solution for a in terms of b.

Expand all terms to produce:

This can now be eliminated into:

Giving a solution for b:

This solution is trivially rewritten as:

Expand the formula for correlation as:

This can now be eliminated into:

Finally, b can be eloquently written as:

Giving a generic formula for the regression line:

Linear Model

The linear model can be expressed as:

If these assumptions can be made:

1. Linearity
2. Exogeneity

3. Random sampling
4. No perfect multicolinearity
5. Heteroskedasticity

Then OLS is the best linear unbiased estimator (BLUE) for these coefficients.

Using the computation above, the coefficients are estimated to produce:

The variance for each coefficient is estimated as:

Where R2 is calculated as:

Note also that the standard deviation of the population's parameter is unknown, so it's estimated like:

CategoryRicottone