Differences between revisions 20 and 33 (spanning 13 versions)
⇤ ← Revision 20 as of 2024-06-05 23:18:46 →
Size: 2060
Editor: DominicRicottone
Comment: Multivariate
← Revision 33 as of 2026-09-03 16:11:49 → ⇥
Size: 3294
Editor: DominicRicottone
Comment: Links
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
'''Ordinary Least Squares''' ('''OLS''') is a linear regression method. It minimizes root mean square errors. '''Ordinary Least Squares''' ('''OLS''') is a linear regression method, and is effectively synonymous with the '''general linear model'''.
Line 11: Line 11:
== Univariate == == Description ==
Line 13: Line 13:
Given one independent variable and one dependent (outcome) variable, the OLS model is specified as: A simple (univariate) linear model is expressed as:
Line 17: Line 17:
It is estimated as: The general linear model (multivariate with ''k'' predictors) is expressed as:

{{attachment:mmodel.svg}}

Note that the predictors are treated as fixed values, not random variables. Only the outcome and [[Analysis/Error|error]] terms are random variables. Measurements of the outcome variable are also available, but the random variable underlying it is modeled as a linear combination of the predictors (i.e., explained variance) and the error term (i.e., unexplained variance). The measurements of the outcome variable are instead used to calculate [[Statistics/Residuals|residuals]]. Further note that, by convention, the error term is represented by a stylized epsilon ''ϵ'' (as opposed to ''ε'').

There are crucial assumptions in this model:
 * independence of all predictors
 * ''ϵ,,i,, ~ N(0, σ^2^)''; which is to say...
   * the error term is a [[Analysis/NormalDistribution|normally distributed]] random variable
   * the [[Analysis/Variance|variance]] of the error term is constant across all cases; ''Var[ϵ,,i,,] = σ^2^'' for all ''i''
   * the [[Analysis/ExpectedValue|expected value]] of the error term is 0; ''E[ϵ,,i,,] = 0'' for all ''i''

These enable the statement that e.g., ''E[y,,i,,|x,,i,,] = α + βx,,i,,''.

This model can be fit using the method of ordinary least squares. And furthermore this method is the '''best''' way to fit the model (under certain assumptions).



=== Single Regression ===

In the case of a single predictor, the OLS regression is:
Line 21: Line 42:
This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable. This formulation leaves the components explicit: the y-intercept term is the mean outcome at ''x=0'', and the slope term is marginal change to the outcome per a unit change in ''x''.
Line 23: Line 44:
The proof can be seen [[Econometrics/OrdinaryLeastSquares/UnivariateProof|here]].

----		 The derivation can be seen [[Statistics/OrdinaryLeastSquares/Single|here]].
Line 29: Line 48:
== Multivariate ==
Line 31: Line 49:
Given ''k'' independent variables, the OLS model is specified as: === Multiple Regression ===
Line 33: Line 51:
{{attachment:mmodel.svg}}

It is estimated as:		 In the case of multiple predictors, the regression is fit like:
Line 39: Line 55:
More conventionally, this is estimated with [[LinearAlgebra|linear algebra]] as: But conventionally, this OLS system is solved using [[LinearAlgebra|linear algebra]] as:
Line 43: Line 59:
The proof can be seen [[Econometrics/OrdinaryLeastSquares/MultivariateProof|here]]. Note that using a ''b'' here is [[Statistics/EconometricsNotation#Models|intentional]].

The derivation can be seen [[Statistics/OrdinaryLeastSquares/Multiple|here]].
Line 51: Line 69:
If these assumptions can be made: The '''Gauss-Markov theorem''' demonstrates that (with some assumptions) the OLS estimations are the '''best linear unbiased estimators''' ('''BLUE''') for the regression coefficients. The assumptions are:
Line 54: Line 72:
 2. [[Econometrics/Exogeneity|Exogeneity]]  2. Exogeneity, i.e. predictors are independent of the outcome and the error term
Line 56: Line 74:
 4. No perfect multicolinearity
 5. [[Econometrics/Homoskedasticity|Homoskedasticity]]		  4. No perfect [[LinearAlgebra/Basis|multicolinearity]]
 5. Homoskedasticity, i.e. error terms are constant across observations
Line 59: Line 77:
Then OLS is the best linear unbiased estimator ('''BLUE''') for regression coefficients.

The variances for each coefficient are:

{{attachment:homo1.svg}}

Note that the standard deviation of the population's parameter is unknown, so it's estimated like:

{{attachment:homo2.svg}}

If the homoskedasticity assumption does not hold, then the estimators for each coefficient are actually:

{{attachment:hetero1.svg}}

Wherein, for example, ''r,,1j,,'' is the residual from regressing ''x,,1,,'' onto ''x,,2,,'', ... ''x,,k,,''.

The variances for each coefficient can be estimated with the Eicker-White formula:

{{attachment:hetero2.svg}}

See [[https://www.youtube.com/@kuminoff|Nicolai Kuminoff's]] video lectures for the derivation of the robust estimators.		 #5 mostly comes into the estimation of [[Statistics/SamplingDistribution|standard errors]], and there are [[Statistics/RobustStandardErrors|alternative estimators that are robust to heteroskedasticity]]]].

Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method, and is effectively synonymous with the general linear model.

Contents

  1. Ordinary Least Squares
    1. Description
      1. Single Regression
      2. Multiple Regression
    2. Estimated Coefficients

Description

A simple (univariate) linear model is expressed as:

model.svg

The general linear model (multivariate with k predictors) is expressed as:

mmodel.svg

Note that the predictors are treated as fixed values, not random variables. Only the outcome and error terms are random variables. Measurements of the outcome variable are also available, but the random variable underlying it is modeled as a linear combination of the predictors (i.e., explained variance) and the error term (i.e., unexplained variance). The measurements of the outcome variable are instead used to calculate residuals. Further note that, by convention, the error term is represented by a stylized epsilon ϵ (as opposed to ε).

There are crucial assumptions in this model:

  • independence of all predictors

  • ϵi ~ N(0, σ2); which is to say...

These enable the statement that e.g., E[yi|xi] = α + βxi.

This model can be fit using the method of ordinary least squares. And furthermore this method is the best way to fit the model (under certain assumptions).

Single Regression

In the case of a single predictor, the OLS regression is:

estimate.svg

This formulation leaves the components explicit: the y-intercept term is the mean outcome at x=0, and the slope term is marginal change to the outcome per a unit change in x.

The derivation can be seen here.

Multiple Regression

In the case of multiple predictors, the regression is fit like:

mestimate.svg

But conventionally, this OLS system is solved using linear algebra as:

matrix.svg

Note that using a b here is intentional.

The derivation can be seen here.

Estimated Coefficients

The Gauss-Markov theorem demonstrates that (with some assumptions) the OLS estimations are the best linear unbiased estimators (BLUE) for the regression coefficients. The assumptions are:

  1. Linearity
  2. Exogeneity, i.e. predictors are independent of the outcome and the error term
  3. Random sampling

  4. No perfect multicolinearity

  5. Homoskedasticity, i.e. error terms are constant across observations

#5 mostly comes into the estimation of standard errors, and there are alternative estimators that are robust to heteroskedasticity]].

CategoryRicottone

Statistics/OrdinaryLeastSquares (last edited 2026-09-03 16:11:49 by DominicRicottone)