Differences between revisions 3 and 18 (spanning 15 versions)

Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method. It minimizes root mean square errors.

Contents

Ordinary Least Squares

Univariate

Given one independent variable and one dependent (outcome) variable, the OLS model is specified as:

It is estimated as:

This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable.

The proof can be seen here.

Multivariate

Linear Model

The linear model can be expressed as:

If these assumptions can be made:

Linearity
Exogeneity
Random sampling
No perfect multicolinearity
Homoskedasticity

Then OLS is the best linear unbiased estimator (BLUE) for these coefficients.

Using the computation above, the coefficients are estimated to produce:

The variances for each coefficient are:

Note that the standard deviation of the population's parameter is unknown, so it's estimated like:

If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:

Wherein, for example, r_1j is the residual from regressing x₁ onto x₂, ... x_k.

The variances for each coefficient can be estimated with the Eicker-White formula:

See Nicolai Kuminoff's video lectures for the derivation of the robust estimators.

CategoryRicottone

Statistics/OrdinaryLeastSquares (last edited 2025-09-03 02:08:40 by DominicRicottone)

-  ⇤ ← Revision 3 as of 2023-10-28 05:37:48 → 
  Size: 1349
  Editor: DominicRicottone
  Comment:
+   ← Revision 18 as of 2024-06-05 21:29:24 → ⇥
  Size: 1866
  Editor: DominicRicottone
  Comment: Rewrite
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-## page was renamed from Econometrics/LinearRegression
-Line 14:
+Line 13:
-The regression line passes through two points:
+Given one independent variable and one dependent (outcome) variable, the OLS model is specified as:
-Line 16:
+Line 15:
-{{attachment:regression1.svg}}
+{{attachment:model.svg}}
-Line 18:
+Line 17:
-and
+It is estimated as:
-Line 20:
+Line 19:
-{{attachment:regression2.svg}}
+{{attachment:estimate.svg}}
-Line 22:
+Line 21:
-Take the generic equation form of a line:
+This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable.
-Line 24:
+Line 23:
-{{attachment:b01.svg}}
+The proof can be seen [[Econometrics/OrdinaryLeastSquares/UnivariateProof|here]].
-Line 26:
+Line 25:
-Insert the first point into this form.
+----
-Line 28:
+Line 27:
-{{attachment:b02.svg}}
-Line 30:
+Line 28:
-This can be trivially rewritten to solve for ''a'' in terms of ''b'':
-Line 32:
+Line 29:
-{{attachment:b03.svg}}
+== Multivariate ==
-Line 34:
+Line 31:
-Insert the second point into the original form.
+----
-Line 36:
+Line 33:
-{{attachment:b04.svg}}
-Line 38:
+Line 34:
-Now additionally insert the solution for ''a'' in terms of ''b''.
-Line 40:
+Line 35:
-{{attachment:b05.svg}}
+== Linear Model ==
-Line 42:
+Line 37:
-Expand all terms to produce:
+The linear model can be expressed as:
-Line 44:
+Line 39:
-{{attachment:b06.svg}}
+{{attachment:model1.svg}}
-Line 46:
+Line 41:
-This can now be eliminated into:
+If these assumptions can be made:
-Line 48:
+Line 43:
-{{attachment:b07.svg}}
+. Linearity
 2. [[Econometrics/Exogeneity|Exogeneity]]
 3. Random sampling
 4. No perfect multicolinearity
 5. [[Econometrics/Homoskedasticity|Homoskedasticity]]
-Line 50:
+Line 49:
-Giving a solution for ''b'':
+Then OLS is the best linear unbiased estimator ('''BLUE''') for these coefficients.
-Line 52:
+Line 51:
-{{attachment:b08.svg}}
+Using the computation above, the coefficients are estimated to produce:
-Line 54:
+Line 53:
-This solution is trivially rewritten as:
+{{attachment:model2.svg}}
-Line 56:
+Line 55:
-{{attachment:b09.svg}}
+The variances for each coefficient are:
-Line 58:
+Line 57:
-Expand the formula for correlation as:
+{{attachment:homo1.svg}}
-Line 60:
+Line 59:
-{{attachment:b10.svg}}
+Note that the standard deviation of the population's parameter is unknown, so it's estimated like:
-Line 62:
+Line 61:
-This can now be eliminated into:
+{{attachment:homo2.svg}}
-Line 64:
+Line 63:
-{{attachment:b11.svg}}
+If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:
-Line 66:
+Line 65:
-Finally, ''b'' can be eloquently written as:
+{{attachment:hetero1.svg}}
-Line 68:
+Line 67:
-{{attachment:b12.svg}}
+Wherein, for example, ''r,,1j,,'' is the residual from regressing ''x,,1,,'' onto ''x,,2,,'', ... ''x,,k,,''.
-Line 70:
+Line 69:
-Giving a generic formula for the regression line:
+The variances for each coefficient can be estimated with the Eicker-White formula:
-Line 72:
+Line 71:
-{{attachment:b13.svg}}
+{{attachment:hetero2.svg}}

See [[https://www.youtube.com/@kuminoff|Nicolai Kuminoff's]] video lectures for the derivation of the robust estimators.

Diff for "Statistics/OrdinaryLeastSquares"

Ordinary Least Squares

Univariate

Multivariate

Linear Model