|
Size: 2578
Comment: Reorganize to vectors and matrices
|
Size: 3042
Comment: Reorganized and added links
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 19: | Line 19: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. | A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. |
| Line 21: | Line 21: |
| Incidentally, '''''P''''' is symmetric (i.e. '''''P'''^T^ = '''P''''') and re-projecting does not change the result (i.e. '''''P'''^2^ = '''P'''''). | Incidentally, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). |
| Line 37: | Line 37: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^. | A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^. |
| Line 39: | Line 39: |
| Note that if '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. This fundamentally means though that if ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. | As above, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). Note that if '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. |
| Line 42: | Line 44: |
Note that if ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. And if ''b'' were [[LinearAlgebra/Orthogonality|orthogonal]] to the column space of '''''A''''', then necessarily ''b'' is the [[LinearAlgebra/NullSpaces|null space]] of '''''A'''^T^'', so '''''P'''b = 0''. |
Projections
When two vectors do not exist in the same column space, the best approximation of one in the other's columns space is called a projection.
Contents
Vectors
Given two vectors a and b, we can project b onto a to get the best possible estimate of the former as a multiple of the latter. This projection p has an error term e.
Take the multiple as x, so that p = ax. The error term can be characterized as b-p or b-ax.
a is orthogonal to e. Therefore, aT(b-ax) = 0. This simplifies to x = (aTb)/(aTa). Altogether, the projection is characterized as p = a(aTb)/(aTa).
A matrix P can be defined such that p = Pb. The projection matrix is (aaT)/(aTa). The column space of P (a.k.a. C(P)) is the line through a, and its rank is 1.
Incidentally, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Matrices
For problems like Ax = b where there is no solution for x, as in b does not exist in the column space of A, we can instead solve Ax = p where p estimates b with an error term e.
p is a linear combination of A: if there are two columns a1 and a2, then p = x1a1 + x2a2 and b = x1a1 + x2a2 + e.
e is orthogonal to the column space of AT (a.k.a. C(AT)), so AT(b-Ax) = 0. Concretely in the same example, a1T(b-Ax) = 0 and a2T(b-Ax) = 0. More generally, this re-emphasizes that e is orthogonal in the null space of AT (a.k.a. N(AT)).
The solution for this all is x = (ATA)-1ATb. That also means that p = A(ATA)-1ATb.
A matrix P can be defined such that p = Pb. The projection matrix is A(ATA)-1AT.
As above, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Note that if A were a square matrix, most of the above equations would cancel out. But we cannot make that assumption.
Note that if b were in the column space of A, then P would be the identity matrix. And if b were orthogonal to the column space of A, then necessarily b is the null space of AT, so Pb = 0.