= Covariance =

'''Covariance''' is an extension of [[Analysis/Variance|variance]].

<<TableOfContents>>

----



== Description ==

Covariance is usually notated as ''Cov(X,Y)'' or ''σ,,X,Y,,''. It is an extension of [[Analysis/Variance|variance]] in the sense that ''Var(X) = Cov(X,X)''. More generally, given random variables ''X'' and ''Y'', their covariance is formulated as ''Cov(X,Y) = E[(X - E[X])(Y - E[Y])]''.

The variance of a random vector (of size ''n x 1'') is a '''covariance matrix''' (of size ''n x n''), usually notated as ''Σ''. Each cell is equal to ''σ,,i,j,,'', where if ''i = j'' then ''σ,,i,j,, = σ,,i,,^2^''.



=== Properties in Bivariate Case ===

In the bivariate case, given random variables ''X'' and ''Y'', adding constants to either random variable does not change the covariance. ''Cov(X+a,Y+b) = Cov(X,Y)''.

Covariance also linearly scales with constant factors:
 * ''Cov(aX,Y) = E[aXY] - E[aX]E[Y]''
 * ''a E[XY] - a E[X]E[Y]''
 * ''a (E[XY] - E[X]E[Y])''
 * ''a Cov(X,Y)''

Recall that ''Var[aX] = a^2^ Var[X]''. Because both variables are scaled by ''a'', the (co)variance is linearly scaled by ''a'' twice.



=== Properties in Multivariate Case ===

Consider now the case of a random vector ''X'' of size ''n x 1''. Adding a constant vector does not change the covariance matrix. ''Var[X + b] = Var[X]''.

Multiplying the random vector by a constant factor scales the known variance by the square of the factor. Succinctly: ''Var[aX] = a^2^ Var[X]''.

Similarly, let ''A'' be a constant matrix of size ''m x n'' representing a linear transformation. It follows that ''Var[AX] = A Var[X] A^T^''.

A covariance matrix is necessarily square, [[LinearAlgebra/SpecialMatrices#Symmetric_Matrices|symmetric]], and [[LinearAlgebra/PositiveDefiniteness|positive semi-definite]]. It follows that:

 * ''Σ = Σ^T^''
 * the [[LinearAlgebra/Determinant|determinant]] is bound by ''|Σ| ≥ 0'' and is called '''generalized variance'''
 * ''Σ^0.5^'' can always be evaluated



=== Estimation ===

When working with a set of paired observations ''X'' and ''Y'', where for each observation ''i'' there are observed values ''x,,i,,'' and ''y,,i,,'', sample covariance is calculated as:

{{attachment:samp.svg}}

When working with a random vector ''x'', with ''μ'' as the corresponding [[Analysis/ExpectedValue|mean vector]], the sample covariance matrix is calculated as:

{{attachment:sampmat.svg}}

----



== Formulation ==

Covariance is an expected value:

{{attachment:exp1.svg}}

By expanding terms and spreading the expected value function, a simpler formula is derived. Note that expectations are linear (i.e., ''E[a + b] = E[a] + E[b]'') and that the expected value of an expected value is the expected value itself (i.e., ''E[E[a]] = E[a]'', so also ''E[E[a]b] = E[a]E[b]''). 

{{attachment:exp2.svg}}

In terms of a random vector ''x'', the covariance matrix is usually expressed as:

{{attachment:exp3.svg}}



----
CategoryRicottone