Projection

A projection is an approximation within a column space.

Vectors

Given two vectors a and b, b can be projected into C(a), the column space of a.

Furthermore, the projection vector with the least error as compared to the true vector b is characterized by orthogonality. Let e be the error vector; it is orthogonal to a.

The transformation of vector b into projection vector p can be described by a projection matrix. It is notated P as in p = Pb.

In Euclidean space, the calculation for a projection vector is is usually expressed as one of:

• p = (a⋅b/||a||) a/||a||

• p = (a⋅b/||a||²) a

• p = (a⋅b/a⋅a) a

But generalizing to inner product spaces, the most common expression is:

Properties

The projection matrix P satisfying p = Pb is rank 1.

C(P), the column space of the projection matrix, is equivalent to C(b).

Projection matrices are symmetric (i.e., P^T = P) and idempotent (i.e., P² = P).

Matrices

For all the same reasons, a vector b can be projected into C(A), the column space of A. The error vector e is orthogonal to R(A), the row space of A, and is therefore in the null space.

The projection matrix is now notated P as in p = Pb.

Least Squares

Given a consistent system as Ax = b, i.e. b is in C(A), there are solutions for x.

If the system is inconsistent, then there is no solution. The best approximation is expressed as Ax̂ = p where projection p estimates b with an error term e. This should sound familiar.

The error term can be generally characterized by e = b - p. An expression for p is known, so e = b - Ax̂.

e is orthogonal to R(A), so A^Te = 0. Substituting in the above expression gives A^T(b - Ax̂) = 0

Altogether, A^TAx̂ = A^Tb which simplifies to x̂ = (A^TA)^-1A^Tb.

The projection matrix is calculated as P = A(A^TA)^-1A^T.

The projection is calculated as p = A(A^TA)^-1A^Tb.

Properties

Projection matrices are still symmetric and idempotent.

If b is in C(A), then P = I.. Conversely, if b is orthogonal to C(A), then Pb = 0 and b = e.

CategoryRicottone