|
Size: 2959
Comment: Initial commit
|
← Revision 23 as of 2026-09-03 16:20:35 ⇥
Size: 4614
Comment: Update content
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Standard Errors = | = Robust Standard Errors = |
| Line 3: | Line 3: |
| '''Standard errors''' are the standard deviations of estimated coefficients. | '''Robust standard errors''' are corrections to classical standard errors to account for failures of assumptions, mainly constant variance across all observations. |
| Line 11: | Line 11: |
| == Description == | == Background == |
| Line 13: | Line 13: |
| In the classical OLS model,, estimated coefficients are: * univariate case: \hat{\beta} = \frac{Cov(X,Y)}{Var(X)} * multivariate case: \mathbf{b} = (\mathbf(X)^T\mathbf(X))^{-1}\mathbf(X)^T\mathbf(y) |
A standard error is the [[Analysis/Variance|variance]] of a [[Statistics/SamplingDistribution|sample statistic]]. |
| Line 17: | Line 15: |
| Standard errors are the standard deviations of these coefficients. | === Classical Standard Error of an Estimated Coefficient in the Simple Linear Regression === In the univariate case, standard errors are classically specified as: {{attachment:unispec1.svg}} Supposing the population ''Var(ε)'' is known and errors are homoskedastic, i.e. they are constant across all cases, this can be simplified. {{attachment:unispec2.svg}} Lastly, rewrite the denominator in terms of ''Var(X)''. {{attachment:unispec3.svg}} ''Var(ε)'' is unknown, so this term is estimated as: {{attachment:uniest2.svg}} {{attachment:uniest1.svg}} 1 degree of freedom is lost in assuming homoskedasticity of errors, i.e. {{attachment:homosked.svg}}; and ''k'' degrees of freedom are lost in assuming independence of errors and ''k'' independent variables, which is necessarily 1 in the univariate case, i.e.: {{attachment:ind.svg}} This arrives at estimation as: {{attachment:uniest3.svg}} === In General Linear Model === The classical multivariate specification is expressed in terms of ''('''b'''-β)'', as: {{attachment:multspec1.svg}} That term is rewritten as ''('''X'''^T^'''X''')^-1^'''Xε'''''. {{attachment:multspec2.svg}} {{attachment:multspec3.svg}} ''E['''εε'''^T^|'''X''']'' is not a practical matrix to work with, even if known. But if homoskedasticity and independence are assumed, i.e.: {{attachment:homosked_ind.svg}}, then this simplifies to: {{attachment:multspec4.svg}} ''s^2^'' is unknown, so this term is estimated as: {{attachment:multspec5.svg}} This arrives at estimation as: {{attachment:multspec6.svg}} |
| Line 23: | Line 73: |
| == Classical == | == Robust == |
| Line 25: | Line 75: |
| === Univariate === | In the presence of heteroskedasticity of errors, the above simplifications cannot apply. In the univariate case, use the original estimator. |
| Line 27: | Line 77: |
| In the univariate case, standard errors are classically specified as: | This is mostly interesting in the multivariate case, where ''E['''εε'''^T^|'''X''']'' is still not practical. The assumptions made, when incorrect, lead to... * OLS estimators are not BLUE * they are unbiased, but no longer most efficient in terms of MSE * nonlinear GLMs, such as logit, can be biased * even if the model's estimates are unbiased, statistics derived from those estimates (e.g., conditioned probability distributions) can be biased |
| Line 29: | Line 83: |
| Var(\hat(\beta)|X_i) = \frac{\sum_{i=1}^n Var((X_i-\bar{X})\hat{\epsilon}_i)}{(\sum_{i=1}^n(X_i-\bar{X})^2)^2} | '''Eicker-Huber-White heterskedasticity consistent errors''' ('''HCE''') assume that errors are still independent but allowed to vary, i.e. '''''Σ''' = diag(ε,,1,,,...ε,,n,,)''. Importantly, this is not a function of '''''X''''', so the standard errors can be estimated as: |
| Line 31: | Line 85: |
| Supposing the population ''Var(β)'' is known, this can be simplified. | {{attachment:robust.svg}} |
| Line 33: | Line 87: |
| Var(\hat{\beta}|X_i) = \frac{Var(\beta)(\sum_{i=1}^n(x_i-\bar{X})^2)}{(\sum_{i=1}^n(X_i-\bar{X})^2)^2} = \frac{Var(\beta)}{\sum_{i=1}^n(X_i-\bar{X})^2} | Robust errors are only appropriate with large sample sizes. |
| Line 35: | Line 89: |
| Lastly, rewrite the denominator in terms of ''Var(X)''. | [[SamplingWeightsAndRegressionAnalysis|When fitting a model using data with survey weights, if those weights are a function of predictors including the dependent variable, then heteroskedastic consistent errors should be used.]] |
| Line 37: | Line 91: |
| Var(\hat{\beta}|X_i) = \frac{Var(\beta)}{n (\frac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2)} = \frac{Var(\beta)}{n Var(X)} | [[HowRobustStandardErrorsExposeMethodologicalProblemsTheyDoNotFix|If a model significantly diverges after introducing robust errors, there is likely a specification error.]] |
| Line 39: | Line 93: |
| ''Var(β)'' is unknown, so this term is estimated as: \hat{\epsilon}_i = Y_i - \hat{Y}_i Var(\hat{\epsilon}) = \frac{1}{n-1}\sum_{i=1}^n(\hat{\epsilon}_i^2) 1 degree of freedom is lost in assuming homoskedasticity of errors, i.e.: \sum_{i=1}^n\hat{\epsilon}_i = 0 ''k'' degrees of freedom are lost in assuming independence of errors and ''k'' independent variables, which is necessarily 1 in the univariate case, i.e.: \sum_{i=1}^nX_i\hat{\epsilon}_i = 0 This arrives at estimation as: \hat{Var}(\hat{\beta}|X_i) = \frac{\frac{1}{n-2}Var(\hat{\epsilon})}{n Var(X)} |
---- |
| Line 59: | Line 97: |
| === Multivariate === | == Clustered == |
| Line 61: | Line 99: |
| The classical multivariate specification is expressed in terms of ''('''b'''-β)'', as: | '''Liang-Zeger clustered robust standard errors''' assume that errors covary within clusters. |
| Line 63: | Line 101: |
| Var(\mathbf{b} | \mathbf{X}) = E\Bigl[(\mathbf{b}-\mathbf{\beta})(\mathbf{b}-\mathbf{\beta})^T \Big| \mathbf{X}\Bigr] | {{attachment:cluster1.svg}} |
| Line 65: | Line 103: |
| That term is rewritten as ''('''X'''^T^'''X''')^-1^'''X'''u''. | where '''''x''',,g,,'' is an ''n,,g,,'' by ''k'' matrix constructed by stacking '''''x''',,i,,'' for all ''i'' belonging to cluster ''g''; and '''''ε''',,g,,'' is an ''n,,g,,'' long vector holding the errors for each cluster ''g''. |
| Line 67: | Line 105: |
| Var(\mathbf{b} | \mathbf{X}) = E\Bigl[\bigl((\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{u}\bigr)\bigl((\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{u}\bigr)^{T} \Big| \mathbf{X}\Bigr] = E\Bigl[(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{u}\mathbf{u}^T\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1} \Big| \mathbf{X}\Bigr] | The estimator becomes: |
| Line 69: | Line 107: |
| Var(\mathbf{b} | \mathbf{X}) = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T E\bigl[\mathbf{u}\mathbf{u}^T\big|\mathbf{X}\bigr]\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1} | {{attachment:cluster2.svg}} |
| Line 71: | Line 109: |
| Practically speaking, ''E['''uu'''^T^|'''X''']'' is never known. But if homoskedasticity and independence are assumed, i.e.: | Clustered standard errors should only be used if the sample design or experimental design call for it. * A complex survey sample design leads to differential sampling errors across strata. * A two-stage sample design leads to differential sampling errors for the SSU within each PSU. * Assignment of an experimental treatment at a grouped level often leads to differential errors across those groups. * For time series evaluation of an experimental treatment that is assigned at the individual level, it is generally recommended to cluster at the individual level. |
| Line 73: | Line 115: |
| E\bigl[\mathbf{u}\mathbf{u}^T\big|\mathbf{X}\bigr] = Var(\mathbf{\beta})\mathbf{I}_n | There are parallels between [[Statistics/FixedEffectsModel|fixed effects]] and clusters, but use of one does not mandate nor conflict with the other. |
| Line 75: | Line 117: |
| ...then this simplifies to: | |
| Line 77: | Line 118: |
| Var(\mathbf{b} | \mathbf{X}) = Var(\mathbf{\beta}) (\mathbf{X}^T\mathbf{X})^{-1} | |
| Line 79: | Line 119: |
| ''Var(β)'' is unknown, so the estimate is: \hat{Var}(\mathbf{b} | \mathbf{X}) = \frac{1}{1-k} \mathbf{u}^T\mathbf{u} (\mathbf{X}^T\mathbf{X})^{-1} |
---- CategoryRicottone |
Robust Standard Errors
Robust standard errors are corrections to classical standard errors to account for failures of assumptions, mainly constant variance across all observations.
Contents
Background
A standard error is the variance of a sample statistic.
Classical Standard Error of an Estimated Coefficient in the Simple Linear Regression
In the univariate case, standard errors are classically specified as:
Supposing the population Var(ε) is known and errors are homoskedastic, i.e. they are constant across all cases, this can be simplified.
Lastly, rewrite the denominator in terms of Var(X).
Var(ε) is unknown, so this term is estimated as:
1 degree of freedom is lost in assuming homoskedasticity of errors, i.e. 
; and k degrees of freedom are lost in assuming independence of errors and k independent variables, which is necessarily 1 in the univariate case, i.e.: 
This arrives at estimation as:
In General Linear Model
The classical multivariate specification is expressed in terms of (b-β), as:
That term is rewritten as (XTX)-1Xε.
E[εεT|X] is not a practical matrix to work with, even if known. But if homoskedasticity and independence are assumed, i.e.: 
, then this simplifies to:
s2 is unknown, so this term is estimated as:
This arrives at estimation as:
Robust
In the presence of heteroskedasticity of errors, the above simplifications cannot apply. In the univariate case, use the original estimator.
This is mostly interesting in the multivariate case, where E[εεT|X] is still not practical. The assumptions made, when incorrect, lead to...
- OLS estimators are not BLUE
- they are unbiased, but no longer most efficient in terms of MSE
- nonlinear GLMs, such as logit, can be biased
- even if the model's estimates are unbiased, statistics derived from those estimates (e.g., conditioned probability distributions) can be biased
Eicker-Huber-White heterskedasticity consistent errors (HCE) assume that errors are still independent but allowed to vary, i.e. Σ = diag(ε1,...εn). Importantly, this is not a function of X, so the standard errors can be estimated as:
Robust errors are only appropriate with large sample sizes.
Clustered
Liang-Zeger clustered robust standard errors assume that errors covary within clusters.
where xg is an ng by k matrix constructed by stacking xi for all i belonging to cluster g; and εg is an ng long vector holding the errors for each cluster g.
The estimator becomes:
Clustered standard errors should only be used if the sample design or experimental design call for it.
- A complex survey sample design leads to differential sampling errors across strata.
- A two-stage sample design leads to differential sampling errors for the SSU within each PSU.
- Assignment of an experimental treatment at a grouped level often leads to differential errors across those groups.
- For time series evaluation of an experimental treatment that is assigned at the individual level, it is generally recommended to cluster at the individual level.
There are parallels between fixed effects and clusters, but use of one does not mandate nor conflict with the other.