Standard Errors

Standard errors are the standard deviations of estimated coefficients.


Description

The standard error of some estimate is the variance of that estimate divided by the square root of the sample size.

One common use of standard errors is to estimate margins of error. For a Bernoulli-distributed variable, the standard error is p(1-p) and is maximized at p=0.5. Therefore a conservative standard error is a function of only the sample size.

Standard errors are also used in interpreting the estimated coefficients of a regression model. As a reminder, by classical OLS, estimated coefficients are:

But specific regressions methods require assumptions about variance. Standard errors in this context are much more complicated.


Classical

Univariate

In the univariate case, standard errors are classically specified as:

unispec1.svg

Supposing the population Var(ε) is known and errors are homoskedastic, i.e. they are constant across all cases, this can be simplified.

unispec2.svg

Lastly, rewrite the denominator in terms of Var(X).

unispec3.svg

Var(ε) is unknown, so this term is estimated as:

uniest1.svg, uniest2.svg

1 degree of freedom is lost in assuming homoskedasticity of errors, i.e. homosked.svg; and k degrees of freedom are lost in assuming independence of errors and k independent variables, which is necessarily 1 in the univariate case, i.e.: ind.svg

This arrives at estimation as:

uniest3.svg

Multivariate

The classical multivariate specification is expressed in terms of (b-β), as:

multspec1.svg

That term is rewritten as (XTX)-1.

multspec2.svg

multspec3.svg

E[εεT|X] is not a practical matrix to work with, even if known. But if homoskedasticity and independence are assumed, i.e.: homosked_ind.svg, then this simplifies to:

multspec4.svg

s2 is unknown, so this term is estimated as:

multspec5.svg

This arrives at estimation as:

multspec6.svg


Robust

In the presence of heteroskedasticity of errors, the above simplifications cannot apply. In the univariate case, use the original estimator.

This is mostly interesting in the multivariate case, where E[εεT|X] is still not practical. The assumptions made, when incorrect, lead to...

Eicker-Huber-White heterskedasticity consistent errors (HCE) assume that errors are still independent but allowed to vary, i.e. Σ = diag(ε1,...εn). Importantly, this is not a function of X, so the standard errors can be estimated as:

robust.svg

Robust errors are only appropriate with large sample sizes.

When fitting a model using data with survey weights, if those weights are a function of predictors including the dependent variable, then heteroskedastic consistent errors should be used.

If a model significantly diverges after introducing robust errors, there is likely a specification error.


Clustered

Liang-Zeger clustered robust standard errors assume that errors covary within clusters.

cluster1.svg

where xg is an ng by k matrix constructed by stacking xi for all i belonging to cluster g; and εg is an ng long vector holding the errors for each cluster g.

The estimator becomes:

cluster2.svg

Clustered standard errors should only be used if the sample design or experimental design call for it.

There are parallels between fixed effects and clusters, but use of one does not mandate nor conflict with the other.


Finite Population Correction

Most formulations of standard errors assume the population is unknown and/or infinite. If the population is finite and the sampling rate is high (above 5%), the standard error is too conservative. The finite population correction (FPC) is an adjustment to correct this:

fpc.svg

Intuitively, the FPC is 0 when n = N because there is no sampling error in a census. FPC approaches 1 when n approaches 0, demonstrating that the factor is meaningless for low sampling rates.


CategoryRicottone

Statistics/StandardErrors (last edited 2025-05-26 21:15:15 by DominicRicottone)