Covariance

Covariance is an extension of variance.

Description

Covariance is usually notated as Cov(X,Y) or σ_X,Y. It is an extension of variance in the sense that Var(X) = Cov(X,X). More generally, given random variables X and Y, their covariance is formulated as Cov(X,Y) = E[(X - E[X])(Y - E[Y])].

The variance of a random vector (of size n x 1) is a covariance matrix (of size n x n), usually notated as Σ. Each cell is equal to σ_i,j, where if i = j then σ_i,j = σ_i².

Properties in Bivariate Case

In the bivariate case, given random variables X and Y, adding constants to either random variable does not change the covariance. Cov(X+a,Y+b) = Cov(X,Y).

Covariance also linearly scales with constant factors:

• Cov(aX,Y) = E[aXY] - E[aX]E[Y]

• a E[XY] - a E[X]E[Y]

• a (E[XY] - E[X]E[Y])

• a Cov(X,Y)

Recall that Var[aX] = a² Var[X]. Because both variables are scaled by a, the (co)variance is linearly scaled by a twice.

Properties in Multivariate Case

Consider now the case of a random vector X of size n x 1. Adding a constant vector does not change the covariance matrix. Var[X + b] = Var[X].

Multiplying the random vector by a constant factor scales the known variance by the square of the factor. Succinctly: Var[aX] = a² Var[X].

Similarly, let A be a constant matrix of size m x n representing a linear transformation. It follows that Var[AX] = A Var[X] A^T.

A covariance matrix is necessarily square, symmetric, and positive semi-definite. It follows that:

• Σ = Σ^T

• the determinant is bound by |Σ| ≥ 0 and is called generalized variance

• Σ^0.5 can always be evaluated

Estimation

When working with a set of paired observations X and Y, where for each observation i there are observed values x_i and y_i, sample covariance is calculated as:

When working with a random vector x, with μ as the corresponding mean vector, the sample covariance matrix is calculated as:

Formulation

Covariance is an expected value:

By expanding terms and spreading the expected value function, a simpler formula is derived. Note that expectations are linear (i.e., E[a + b] = E[a] + E[b]) and that the expected value of an expected value is the expected value itself (i.e., E[E[a]] = E[a], so also E[E[a]b] = E[a]E[b]).

In terms of a random vector x, the covariance matrix is usually expressed as:

CategoryRicottone