Differences between revisions 6 and 26 (spanning 20 versions)
⇤ ← Revision 6 as of 2023-10-28 07:04:18 →
Size: 2049
Editor: DominicRicottone
Comment:
← Revision 26 as of 2025-05-17 03:48:23 → ⇥
Size: 1689
Editor: DominicRicottone
Comment: Clarifications
Deletions are marked like this. Additions are marked like this.
Line 13: Line 13:
The regression line passes through two points: Given one independent variable and one dependent (outcome) variable, the OLS model is specified as:
Line 15: Line 15:
{{attachment:regression1.svg}} {{attachment:model.svg}}
Line 17: Line 17:
and It is estimated as:
Line 19: Line 19:
{{attachment:regression2.svg}} {{attachment:estimate.svg}}
Line 21: Line 21:
Take the generic equation form of a line: This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable.
Line 23: Line 23:
{{attachment:b01.svg}}

Insert the first point into this form.

{{attachment:b02.svg}}

This can be trivially rewritten to solve for ''a'' in terms of ''b'':

{{attachment:b03.svg}}

Insert the second point into the original form.

{{attachment:b04.svg}}

Now additionally insert the solution for ''a'' in terms of ''b''.

{{attachment:b05.svg}}

Expand all terms to produce:

{{attachment:b06.svg}}

This can now be eliminated into:

{{attachment:b07.svg}}

Giving a solution for ''b'':

{{attachment:b08.svg}}

This solution is trivially rewritten as:

{{attachment:b09.svg}}

Expand the formula for correlation as:

{{attachment:b10.svg}}

This can now be eliminated into:

{{attachment:b11.svg}}

Finally, ''b'' can be eloquently written as:

{{attachment:b12.svg}}

Giving a generic formula for the regression line:

{{attachment:b13.svg}}		 The derivation can be seen [[Statistics/OrdinaryLeastSquares/Univariate|here]].
Line 77: Line 29:
== Linear Model == == Multivariate ==
Line 79: Line 31:
The linear model can be expressed as: Given ''k'' independent variables, the OLS model is specified as:
Line 81: Line 33:
{{attachment:model1.svg}} {{attachment:mmodel.svg}}
Line 83: Line 35:
If these assumptions can be made: It is estimated as:

{{attachment:mestimate.svg}}

More conventionally, this is estimated with [[LinearAlgebra|linear algebra]] as:

{{attachment:matrix.svg}}

The derivation can be seen [[Statistics/OrdinaryLeastSquares/Multivariate|here]].

----



== Estimated Coefficients ==

The '''Gauss-Markov theorem''' demonstrates that (with some assumptions) the OLS estimations are the '''best linear unbiased estimators''' ('''BLUE''') for the regression coefficients. The assumptions are:
Line 86: Line 54:
 2. Exogeneity  2. Exogeneity, i.e. predictors are independent of the outcome and the error term
 3. Random sampling
 4. No perfect [[LinearAlgebra/Basis|multicolinearity]]
 5. Homoskedasticity, i.e. error terms are constant across observations
Line 88: Line 59:
{{attachment:model2.svg}}

 3.#3 Random sampling
 4. No perfect multicolinearity
 5. Heteroskedasticity

Then OLS is the best linear unbiased estimator ('''BLUE''') for these coefficients.

Using the computation above, the coefficients are estimated to produce:

{{attachment:model3.svg}}

The variance for each coefficient is estimated as:

{{attachment:model4.svg}}

Where R^2^ is calculated as:

{{attachment:model5.svg}}

Note also that the standard deviation of the population's parameter is unknown, so it's estimated like:

{{attachment:model6.svg}}		 #5 mostly comes into the estimation of [[Statistics/StandardErrors|standard errors]], and there are alternative estimators that are robust to heteroskedasticity.

Ordinary Least Squares

Ordinary Least Squares (OLS) is a linear regression method. It minimizes root mean square errors.

Contents

  1. Ordinary Least Squares
    1. Univariate
    2. Multivariate
    3. Estimated Coefficients

Univariate

Given one independent variable and one dependent (outcome) variable, the OLS model is specified as:

model.svg

It is estimated as:

estimate.svg

This model describes (1) the mean observation and (2) the marginal changes to the outcome per unit changes in the independent variable.

The derivation can be seen here.

Multivariate

Given k independent variables, the OLS model is specified as:

mmodel.svg

It is estimated as:

mestimate.svg

More conventionally, this is estimated with linear algebra as:

matrix.svg

The derivation can be seen here.

Estimated Coefficients

The Gauss-Markov theorem demonstrates that (with some assumptions) the OLS estimations are the best linear unbiased estimators (BLUE) for the regression coefficients. The assumptions are:

  1. Linearity
  2. Exogeneity, i.e. predictors are independent of the outcome and the error term
  3. Random sampling

  4. No perfect multicolinearity

  5. Homoskedasticity, i.e. error terms are constant across observations

#5 mostly comes into the estimation of standard errors, and there are alternative estimators that are robust to heteroskedasticity.

CategoryRicottone

Statistics/OrdinaryLeastSquares (last edited 2025-09-03 02:08:40 by DominicRicottone)