Moments

Moments are measures of a distribution's shape and density.

Description

The first raw moment is the mean: μ = E[X]. For discrete variables, this is calculated as Σ x P(x=X); for continuous variables, as ∫ x f(x) dx

The second central moment is the variance: σ² = E[(X - E[X])²] = E[(X - μ)²] = E(X²) - (E[X])²

The derivation of this for discrete variables is:

• Σ (x - μ)² P(x=X)

• Σ (x² - 2μx + μ²) P(x=X)

• Σ [x² P(x=X)] - 2μ Σ [x P(x=X)] + μ² Σ [P(x=X)]

• [E[X²]] - 2μ [μ] + μ² [1]

• E[X²] - 2μ² + μ²

• E[X²] - μ²

• E[X²] - (E[X])²

The derivation of this for continuous variables is:

• ∫ (x - μ)² f(x) dx

• ∫ (x² - 2μx + μ²) f(x) dx

• ∫ [x² f(x) dx] - 2μ ∫ [x f(x) dx] + μ² ∫ [f(x) dx]

• [E[X²]] - 2μ [μ] + μ² [1]

• E[X²] - 2μ² + μ²

• E[X²] - μ²

• E[X²] - (E[X])²

Through these derivations, it can be easily proven that (1) constants added to a variable do not affect variance, and (2) constant multipliers applied to a variable scale variance by their square. This is succinctly summarized as Var(aX + b) = a² Var(X)

The third central moment, skewness, measures lopsidedness of a distribution.

The fourth central moment, kurtosis, measures the heaviness of the tails on a distribution.

Errors

Models generally assume that individual errors average to zero, i.e. the first moment of errors is zero: E[Ŷ - Y] = 0. Nonetheless, higher order moments are important.

The mean square error (MSE) is the second moment of the error: MSE(ˆθ) = E[(ˆθ - E[ˆθ])²]. MSE can be decomposed into the variance of the estimator and bias: MSE(ˆθ) = Var(ˆθ) + Bias(ˆθ,θ)² = Var(ˆθ) + (E[ˆθ]-θ)².

Two important notes:

• Bias, i.e. E[ˆθ] - θ, is not the same as the first moment of errors.

• If there is no bias, then MSE is the variance of the estimator: MSE(ˆθ) = Var(ˆθ).

CategoryRicottone