Differences between revisions 1 and 14 (spanning 13 versions)

Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method. It minimizes root mean square errors.

Contents

Ordinary Least Squares
1. Univariate
2. Linear Model

Univariate

The regression line passes through two points:

and

These points, with the generic equation for a line, can prove that the slope of the regression line is equal to:

The generic formula for the regression line is:

Linear Model

The linear model can be expressed as:

If these assumptions can be made:

Linearity
Exogeneity
Random sampling
No perfect multicolinearity
Homoskedasticity

Then OLS is the best linear unbiased estimator (BLUE) for these coefficients.

Using the computation above, the coefficients are estimated to produce:

The variances for each coefficient are:

Note that the standard deviation of the population's parameter is unknown, so it's estimated like:

If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:

Wherein, for example, r_1j is the residual from regressing x₁ onto x₂, ... x_k.

The variances for each coefficient can be estimated with the Eicker-White formula:

CategoryRicottone

Statistics/OrdinaryLeastSquares (last edited 2025-01-10 14:33:38 by DominicRicottone)

-  ⇤ ← Revision 1 as of 2023-10-28 05:18:15 → 
  Size: 1390
  Editor: DominicRicottone
  Comment:
+   ← Revision 14 as of 2023-10-28 18:20:36 → ⇥
  Size: 1696
  Editor: DominicRicottone
  Comment: Updating heteroskedasticity
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-= Linear Regression =
+= Ordinary Least Squares =
 Line 3:
-A linear regression expresses the linear relation of a treatment variable to an outcome variable.
+'''Ordinary Least Squares''' ('''OLS''') is a linear regression method. It minimizes root mean square errors.
 Line 11:
-== Regression Line ==

A regression line can be especially useful on a scatter plot.
+== Univariate ==
-Line 23:
+Line 21:
+These points, with the generic equation for a line, can [[Econometrics/OrdinaryLeastSquares/UnivariateProof|prove]] that the slope of the regression line is equal to:

{{attachment:b12.svg}}

The generic formula for the regression line is:

{{attachment:b13.svg}}
-Line 27:
+Line 33:
-== Regression Computation ==
+== Linear Model ==
-Line 29:
+Line 35:
-Take the generic equation form of a line:
+The linear model can be expressed as:
-Line 31:
+Line 37:
-{{attachment:b01.svg}}
+{{attachment:model1.svg}}
-Line 33:
+Line 39:
-Insert the first point into this form.
+If these assumptions can be made:
-Line 35:
+Line 41:
-{{attachment:b02.svg}}
+. Linearity
 2. [[Econometrics/Exogeneity|Exogeneity]]
 3. Random sampling
 4. No perfect multicolinearity
 5. [[Econometrics/Homoskedasticity|Homoskedasticity]]
-Line 37:
+Line 47:
-This can be trivially rewritten to solve for ''a'' in terms of ''b'':
+Then OLS is the best linear unbiased estimator ('''BLUE''') for these coefficients.
-Line 39:
+Line 49:
-{{attachment:b03.svg}}
+Using the computation above, the coefficients are estimated to produce:
-Line 41:
+Line 51:
-Insert the second point into the original form.
+{{attachment:model2.svg}}
-Line 43:
+Line 53:
-{{attachment:b04.svg}}
+The variances for each coefficient are:
-Line 45:
+Line 55:
-Now additionally insert the solution for ''a'' in terms of ''b''.
+{{attachment:homo1.svg}}
-Line 47:
+Line 57:
-{{attachment:b05.svg}}
+Note that the standard deviation of the population's parameter is unknown, so it's estimated like:
-Line 49:
+Line 59:
-Expand all terms to produce:
+{{attachment:homo2.svg}}
-Line 51:
+Line 61:
-{{attachment:b06.svg}}
+If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:
-Line 53:
+Line 63:
-This can now be eliminated into:
+{{attachment:hetero1.svg}}
-Line 55:
+Line 65:
-{{attachment:b07.svg}}
+Wherein, for example, ''r,,1j,,'' is the residual from regressing ''x,,1,,'' onto ''x,,2,,'', ... ''x,,k,,''.
-Line 57:
+Line 67:
-Giving a solution for ''b'':
+The variances for each coefficient can be estimated with the Eicker-White formula:
-Line 59:
+Line 69:
-{{attachment:b08.svg}}

This solution is trivially rewritten as:

{{attachment:b09.svg}}

Expand the formula for correlation as:

{{attachment:b10.svg}}

This can now be eliminated into:

{{attachment:b11.svg}}

Finally, ''b'' can be eloquently written as:

{{attachment:b12.svg}}

Giving a generic formula for the regression line:

{{attachment:b13.svg}}
+{{attachment:hetero2.svg}}

Diff for "Statistics/OrdinaryLeastSquares"

Ordinary Least Squares

Univariate

Linear Model