|
Size: 2578
Comment: Reorganize to vectors and matrices
|
Size: 3515
Comment: Reorganized
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 19: | Line 19: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. | A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. |
| Line 21: | Line 21: |
| Incidentally, '''''P''''' is symmetric (i.e. '''''P'''^T^ = '''P''''') and re-projecting does not change the result (i.e. '''''P'''^2^ = '''P'''''). | Incidentally, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). |
| Line 33: | Line 33: |
| ''e'' is orthogonal to the column space of '''''A'''^T^'' (a.k.a. ''C('''A'''^T^)''), so '''''A'''^T^(b-'''A'''x) = 0''. Concretely in the same example, ''a,,1,,^T^(b-'''A'''x) = 0'' and ''a,,2,,^T^(b-'''A'''x) = 0''. More generally, this re-emphasizes that ''e'' is orthogonal in the null space of '''''A'''^T^'' (a.k.a. ''N('''A'''^T^)''). | ''e'' is [[LinearAlgebra/Orthogonality|orthogonal]] to the row space of '''''A''''' because the error term does not exist in any linear combination of the rows. The projection is more easily worked with in terms of '''''A'''^T^'', so instead think of ''e'' being orthogonal to the column space of '''''A'''^T^''. Therefore, '''''A'''^T^(b-'''A'''x) = 0''. Concretely in the same example, ''a,,1,,^T^(b-'''A'''x) = 0'' and ''a,,2,,^T^(b-'''A'''x) = 0''. More generally, that re-emphasizes that ''e'' is the [[LinearAlgebra/NullSpaces|null space]] of '''''A'''^T^''. |
| Line 37: | Line 37: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^. | A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^. |
| Line 39: | Line 39: |
| Note that if '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. This fundamentally means though that if ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. | ''b'' can also be projected onto ''e'', which geometrically means projecting into the null space of '''''A'''^T^''. Algebraically, that projection matrix in terms of '''''P''''' is ''('''I'''-'''P''')b''. As above, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). Note that if '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. |
| Line 42: | Line 46: |
Note that if ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. And if ''b'' were orthogonal to the column space of '''''A''''', then necessarily ''b'' is in the null space of '''''A'''^T^''. For that reason, projecting ''b'' onto ''e'' would give an identity matrix. In that case, '''''P'''b = 0'' and ''b = e''. |
Projections
When two vectors do not exist in the same column space, the best approximation of one in the other's columns space is called a projection.
Contents
Vectors
Given two vectors a and b, we can project b onto a to get the best possible estimate of the former as a multiple of the latter. This projection p has an error term e.
Take the multiple as x, so that p = ax. The error term can be characterized as b-p or b-ax.
a is orthogonal to e. Therefore, aT(b-ax) = 0. This simplifies to x = (aTb)/(aTa). Altogether, the projection is characterized as p = a(aTb)/(aTa).
A matrix P can be defined such that p = Pb. The projection matrix is (aaT)/(aTa). The column space of P (a.k.a. C(P)) is the line through a, and its rank is 1.
Incidentally, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Matrices
For problems like Ax = b where there is no solution for x, as in b does not exist in the column space of A, we can instead solve Ax = p where p estimates b with an error term e.
p is a linear combination of A: if there are two columns a1 and a2, then p = x1a1 + x2a2 and b = x1a1 + x2a2 + e.
e is orthogonal to the row space of A because the error term does not exist in any linear combination of the rows. The projection is more easily worked with in terms of AT, so instead think of e being orthogonal to the column space of AT. Therefore, AT(b-Ax) = 0. Concretely in the same example, a1T(b-Ax) = 0 and a2T(b-Ax) = 0. More generally, that re-emphasizes that e is the null space of AT.
The solution for this all is x = (ATA)-1ATb. That also means that p = A(ATA)-1ATb.
A matrix P can be defined such that p = Pb. The projection matrix is A(ATA)-1AT.
b can also be projected onto e, which geometrically means projecting into the null space of AT. Algebraically, that projection matrix in terms of P is (I-P)b.
As above, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Note that if A were a square matrix, most of the above equations would cancel out. But we cannot make that assumption.
Note that if b were in the column space of A, then P would be the identity matrix. And if b were orthogonal to the column space of A, then necessarily b is in the null space of AT. For that reason, projecting b onto e would give an identity matrix. In that case, Pb = 0 and b = e.