Projection
A projection is an approximation within a column space. A projection matrix describes the linear transformation of the approximation.
See also vector projection.
Contents
Vectors
Given vectors a and b, b can be projected into the column space of a (i.e., C(a)). A projection matrix describes the linear transformation from vector a to projection vector p. Such a projection matrix P satisfies p = Pa.
The column space of the projection matrix (i.e., C(P)) is equivalent to C(b). It follows that P is also of rank 1.
Properties
Projection matrices are symmetric (i.e., PT = P) and idempotent (i.e., P2 = P).
Matrices
Given vector b and matrix A, b can be projected into the column space of A (i.e., C(A)).
Given a linear system as Ax = b, if b is in C(A), there are solutions for x. If b is not in C(A) however, there is no possible solution. The best approximation is expressed as Ax̂ = p where projection p estimates b with an error term e. This should sound familiar.
The error term can be characterized by e = b - p or e = b - Ax̂. e is orthogonal to R(A), the row space of A; equivalently it is orthogonal to C(AT). Orthogonality in this context means that e is in the null space, so AT(b - Ax̂) = 0.
The system of normal equations is ATAx̂ = ATb. This simplifies to x̂ = (ATA)-1ATb. Altogether, the projection is characterized by p = A(ATA)-1ATb.
The projection matrix P satisfies p = Pb. It is calculated as P = A(ATA)-1AT.
b can also be projected onto e, which geometrically means projecting into the null space of AT. Algebraically, if one projection matrix has been computed as P, then the projection matrix for going the other way is (I - P)b.
Properties
As above, the projection matrix P is symmetric and idempotent.
If A is square, the above equations simplify rapidly.
If b is in C(A), then P = I. Conversely, if b is orthogonal to C(A), then Pb = 0 and b = e.