Differences between revisions 1 and 6 (spanning 5 versions)

Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method. It minimizes root mean square errors.

Contents

Univariate

The regression line passes through two points:

and

Take the generic equation form of a line:

Insert the first point into this form.

This can be trivially rewritten to solve for a in terms of b:

Insert the second point into the original form.

Now additionally insert the solution for a in terms of b.

Expand all terms to produce:

This can now be eliminated into:

Giving a solution for b:

This solution is trivially rewritten as:

Expand the formula for correlation as:

This can now be eliminated into:

Finally, b can be eloquently written as:

Giving a generic formula for the regression line:

The linear model can be expressed as:

If these assumptions can be made:

Then OLS is the best linear unbiased estimator (BLUE) for these coefficients.

Using the computation above, the coefficients are estimated to produce:

The variance for each coefficient is estimated as:

Where R² is calculated as:

Note also that the standard deviation of the population's parameter is unknown, so it's estimated like:

Statistics/OrdinaryLeastSquares (last edited 2025-01-10 14:33:38 by DominicRicottone)

-  ⇤ ← Revision 1 as of 2023-10-28 05:18:15 → 
  Size: 1390
  Editor: DominicRicottone
  Comment:
+   ← Revision 6 as of 2023-10-28 07:04:18 → ⇥
  Size: 2049
  Editor: DominicRicottone
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-= Linear Regression =
+= Ordinary Least Squares =
 Line 3:
-A linear regression expresses the linear relation of a treatment variable to an outcome variable.
+'''Ordinary Least Squares''' ('''OLS''') is a linear regression method. It minimizes root mean square errors.
 Line 11:
-== Regression Line ==

A regression line can be especially useful on a scatter plot.
+== Univariate ==
-Line 22:
+Line 20:
-----



== Regression Computation ==
-Line 81:
+Line 73:
+----



== Linear Model ==

The linear model can be expressed as:

{{attachment:model1.svg}}

If these assumptions can be made:

 1. Linearity
 2. Exogeneity

{{attachment:model2.svg}}

 3.#3 Random sampling
 4. No perfect multicolinearity
 5. Heteroskedasticity

Then OLS is the best linear unbiased estimator ('''BLUE''') for these coefficients.

Using the computation above, the coefficients are estimated to produce:

{{attachment:model3.svg}}

The variance for each coefficient is estimated as:

{{attachment:model4.svg}}

Where R^2^ is calculated as:

{{attachment:model5.svg}}

Note also that the standard deviation of the population's parameter is unknown, so it's estimated like:

{{attachment:model6.svg}}