Differences between revisions 3 and 23 (spanning 20 versions)

Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method. It minimizes root mean square errors.

Contents

Ordinary Least Squares

Univariate

Given one independent variable and one dependent (outcome) variable, the OLS model is specified as:

It is estimated as:

This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable.

The derivation can be seen here.

Multivariate

Given k independent variables, the OLS model is specified as:

It is estimated as:

More conventionally, this is estimated with linear algebra as:

The derivation can be seen here.

Estimated Coefficients

The Gauss-Markov theorem demonstrates that (with some assumptions) the OLS estimations are the best linear unbiased estimators (BLUE) for the regression coefficients. The assumptions are:

Linearity
Exogeneity
Random sampling
No perfect multicolinearity
Homoskedasticity

The variances for each coefficient are:

Note that the standard deviation of the population's parameter is unknown, so it's estimated like:

If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:

Wherein, for example, r_1j is the residual from regressing x₁ onto x₂, ... x_k.

The variances for each coefficient can be estimated with the Eicker-White formula:

See Nicolai Kuminoff's video lectures for the derivation of the robust estimators.

CategoryRicottone

Statistics/OrdinaryLeastSquares (last edited 2025-09-03 02:08:40 by DominicRicottone)

-  ⇤ ← Revision 3 as of 2023-10-28 05:37:48 → 
  Size: 1349
  Editor: DominicRicottone
  Comment:
+   ← Revision 23 as of 2025-01-10 14:33:11 → ⇥
  Size: 2224
  Editor: DominicRicottone
  Comment: Killing Econometrics page
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
-## page was renamed from Econometrics/LinearRegression
+## page was renamed from Econometrics/OrdinaryLeastSquares
 Line 14:
-The regression line passes through two points:
+Given one independent variable and one dependent (outcome) variable, the OLS model is specified as:
 Line 16:
-{{attachment:regression1.svg}}
+{{attachment:model.svg}}
 Line 18:
-and
+It is estimated as:
 Line 20:
-{{attachment:regression2.svg}}
+{{attachment:estimate.svg}}
 Line 22:
-Take the generic equation form of a line:
+This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable.
 Line 24:
-{{attachment:b01.svg}}
+The derivation can be seen [[Econometrics/OrdinaryLeastSquares/Univariate|here]].
 Line 26:
-Insert the first point into this form.
+----
 Line 28:
-{{attachment:b02.svg}}
-Line 30:
+Line 29:
-This can be trivially rewritten to solve for ''a'' in terms of ''b'':
-Line 32:
+Line 30:
-{{attachment:b03.svg}}
+== Multivariate ==
-Line 34:
+Line 32:
-Insert the second point into the original form.
+Given ''k'' independent variables, the OLS model is specified as:
-Line 36:
+Line 34:
-{{attachment:b04.svg}}
+{{attachment:mmodel.svg}}
-Line 38:
+Line 36:
-Now additionally insert the solution for ''a'' in terms of ''b''.
+It is estimated as:
-Line 40:
+Line 38:
-{{attachment:b05.svg}}
+{{attachment:mestimate.svg}}
-Line 42:
+Line 40:
-Expand all terms to produce:
+More conventionally, this is estimated with [[LinearAlgebra|linear algebra]] as:
-Line 44:
+Line 42:
-{{attachment:b06.svg}}
+{{attachment:matrix.svg}}
-Line 46:
+Line 44:
-This can now be eliminated into:
+The derivation can be seen [[Econometrics/OrdinaryLeastSquares/Multivariate|here]].
-Line 48:
+Line 46:
-{{attachment:b07.svg}}
+----
-Line 50:
+Line 48:
-Giving a solution for ''b'':
-Line 52:
+Line 49:
-{{attachment:b08.svg}}
-Line 54:
+Line 50:
-This solution is trivially rewritten as:
+== Estimated Coefficients ==
-Line 56:
+Line 52:
-{{attachment:b09.svg}}
+The '''Gauss-Markov theorem''' demonstrates that (with some assumptions) the OLS estimations are the '''best linear unbiased estimators''' ('''BLUE''') for the regression coefficients. The assumptions are:
-Line 58:
+Line 54:
-Expand the formula for correlation as:
+. Linearity
 2. [[Econometrics/Exogeneity|Exogeneity]]
 3. Random sampling
 4. No perfect [[LinearAlgebra/Basis|multicolinearity]]
 5. [[Econometrics/Homoskedasticity|Homoskedasticity]]
 Line 60:
-{{attachment:b10.svg}}
+The variances for each coefficient are:
 Line 62:
-This can now be eliminated into:
+{{attachment:homo1.svg}}
 Line 64:
-{{attachment:b11.svg}}
+Note that the standard deviation of the population's parameter is unknown, so it's estimated like:
 Line 66:
-Finally, ''b'' can be eloquently written as:
+{{attachment:homo2.svg}}
 Line 68:
-{{attachment:b12.svg}}
+If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:
 Line 70:
-Giving a generic formula for the regression line:
+{{attachment:hetero1.svg}}
 Line 72:
-{{attachment:b13.svg}}
+Wherein, for example, ''r,,1j,,'' is the residual from regressing ''x,,1,,'' onto ''x,,2,,'', ... ''x,,k,,''.

The variances for each coefficient can be estimated with the Eicker-White formula:

{{attachment:hetero2.svg}}

See [[https://www.youtube.com/@kuminoff|Nicolai Kuminoff's]] video lectures for the derivation of the robust estimators.

Diff for "Statistics/OrdinaryLeastSquares"

Ordinary Least Squares

Univariate

Multivariate

Estimated Coefficients