Projections
When two vectors do not exist in the same column space, the best approximation of one in the other's columns space is called a projection.
Contents
Vectors
Given two vectors a and b, we can project b onto a to get the best possible estimate of the former as a multiple of the latter. This projection p has an error term e.
Take the multiple as x, so that p = ax. The error term can be characterized as b-p or b-ax.
a is orthogonal to e. Therefore, aT(b-ax) = 0. This simplifies to x = (aTb)/(aTa). Altogether, the projection is characterized as p = a(aTb)/(aTa).
A matrix P can be defined such that p = Pb. The projection matrix is (aaT)/(aTa). The column space of P (a.k.a. C(P)) is the line through a, and its rank is 1.
Incidentally, P is symmetric (i.e. PT = P) and re-projecting does not change the result (i.e. P2 = P).
Matrices
For problems like Ax = b where there is no solution for x, as in b does not exist in the column space of A, we can instead solve Ax = p where p estimates b with an error term e.
p is a linear combination of A: if there are two columns a1 and a2, then p = x1a1 + x2a2 and b = x1a1 + x2a2 + e.
e is orthogonal to the column space of AT (a.k.a. C(AT)), so AT(b-Ax) = 0. Concretely in the same example, a1T(b-Ax) = 0 and a2T(b-Ax) = 0. More generally, this re-emphasizes that e is orthogonal in the null space of AT (a.k.a. N(AT)).
The solution for this all is x = (ATA)-1ATb. That also means that p = A(ATA)-1ATb.
A matrix P can be defined such that p = Pb. The projection matrix is A(ATA)-1AT.
Note that if A were a square matrix, most of the above equations would cancel out. But we cannot make that assumption. This fundamentally means though that if b were in the column space of A, then P would be the identity matrix.