|
⇤ ← Revision 1 as of 2024-01-21 02:40:01
Size: 729
Comment: Initial commit
|
Size: 2578
Comment: Reorganize to vectors and matrices
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
When two vectors do not exist in the same column space, the best approximation of one in the other's columns space is called a '''projection'''. |
|
| Line 9: | Line 11: |
| == Definition == | == Vectors == |
| Line 13: | Line 15: |
| Take the multiple as ''x'', so that ''p = ax''. The error term can be characterized as ''b-ax''. | Take the multiple as ''x'', so that ''p = ax''. The error term can be characterized as ''b-p'' or ''b-ax''. |
| Line 17: | Line 19: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is ''(aa^T^)/(a^T^a)''. | A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. Incidentally, '''''P''''' is symmetric (i.e. '''''P'''^T^ = '''P''''') and re-projecting does not change the result (i.e. '''''P'''^2^ = '''P'''''). ---- == Matrices == For problems like '''''A'''x = b'' where there is no solution for ''x'', as in b does not exist in the column space of '''''A''''', we can instead solve '''''A'''x = p'' where ''p'' estimates ''b'' with an error term ''e''. ''p'' is a linear combination of '''''A''''': if there are two columns ''a,,1,,'' and ''a,,2,,'', then ''p = x,,1,,a,,1,, + x,,2,,a,,2,,'' and ''b = x,,1,,a,,1,, + x,,2,,a,,2,, + e''. ''e'' is orthogonal to the column space of '''''A'''^T^'' (a.k.a. ''C('''A'''^T^)''), so '''''A'''^T^(b-'''A'''x) = 0''. Concretely in the same example, ''a,,1,,^T^(b-'''A'''x) = 0'' and ''a,,2,,^T^(b-'''A'''x) = 0''. More generally, this re-emphasizes that ''e'' is orthogonal in the null space of '''''A'''^T^'' (a.k.a. ''N('''A'''^T^)''). The solution for this all is ''x = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. That also means that ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^. Note that if '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. This fundamentally means though that if ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. [[Econometrics/OrdinaryLeastSquares|This should look familiar.]] |
Projections
When two vectors do not exist in the same column space, the best approximation of one in the other's columns space is called a projection.
Contents
Vectors
Given two vectors a and b, we can project b onto a to get the best possible estimate of the former as a multiple of the latter. This projection p has an error term e.
Take the multiple as x, so that p = ax. The error term can be characterized as b-p or b-ax.
a is orthogonal to e. Therefore, aT(b-ax) = 0. This simplifies to x = (aTb)/(aTa). Altogether, the projection is characterized as p = a(aTb)/(aTa).
A matrix P can be defined such that p = Pb. The projection matrix is (aaT)/(aTa). The column space of P (a.k.a. C(P)) is the line through a, and its rank is 1.
Incidentally, P is symmetric (i.e. PT = P) and re-projecting does not change the result (i.e. P2 = P).
Matrices
For problems like Ax = b where there is no solution for x, as in b does not exist in the column space of A, we can instead solve Ax = p where p estimates b with an error term e.
p is a linear combination of A: if there are two columns a1 and a2, then p = x1a1 + x2a2 and b = x1a1 + x2a2 + e.
e is orthogonal to the column space of AT (a.k.a. C(AT)), so AT(b-Ax) = 0. Concretely in the same example, a1T(b-Ax) = 0 and a2T(b-Ax) = 0. More generally, this re-emphasizes that e is orthogonal in the null space of AT (a.k.a. N(AT)).
The solution for this all is x = (ATA)-1ATb. That also means that p = A(ATA)-1ATb.
A matrix P can be defined such that p = Pb. The projection matrix is A(ATA)-1AT.
Note that if A were a square matrix, most of the above equations would cancel out. But we cannot make that assumption. This fundamentally means though that if b were in the column space of A, then P would be the identity matrix.