Projections
When two vectors do not exist in the same column space, the best approximation of one in the other's columns space is called a projection.
Contents
Vectors
Given two vectors a and b, we can project b onto a to get the best possible estimate of the former as a multiple of the latter. This projection p has an error term e.
Take the multiple as x, so that p = ax. The error term can be characterized as b-p or b-ax.
a is orthogonal to e. Therefore, aT(b-ax) = 0. This simplifies to x = (aTb)/(aTa). Altogether, the projection is characterized as p = a(aTb)/(aTa).
A matrix P can be defined such that p = Pb. The projection matrix is (aaT)/(aTa). The column space of P (a.k.a. C(P)) is the line through a, and its rank is 1.
Incidentally, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Matrices
For problems like Ax = b where there is no solution for x, as in b does not exist in the column space of A, we can instead solve Ax = p where p estimates b with an error term e.
p is a linear combination of A: if there are two columns a1 and a2, then p = x1a1 + x2a2 and b = x1a1 + x2a2 + e.
e is orthogonal to the row space of A because the error term does not exist in any linear combination of the rows. The projection is more easily worked with in terms of AT, so instead think of e being orthogonal to the column space of AT. Therefore, AT(b-Ax) = 0. Concretely in the same example, a1T(b-Ax) = 0 and a2T(b-Ax) = 0. More generally, that re-emphasizes that e is the null space of AT.
The solution for this all is x = (ATA)-1ATb. That also means that p = A(ATA)-1ATb.
A matrix P can be defined such that p = Pb. The projection matrix is A(ATA)-1AT.
b can also be projected onto e, which geometrically means projecting into the null space of AT. Algebraically, that projection matrix in terms of P is (I-P)b.
As above, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Note that if A were a square matrix, most of the above equations would cancel out. But we cannot make that assumption.
Note that if b were in the column space of A, then P would be the identity matrix. And if b were orthogonal to the column space of A, then necessarily b is in the null space of AT. For that reason, projecting b onto e would give an identity matrix. In that case, Pb = 0 and b = e.