|
Size: 729
Comment: Initial commit
|
Size: 3609
Comment: Updated intro
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
When a vector does not exist in s column space, the best approximation of it in that columns space is called a '''projection'''. |
|
| Line 9: | Line 11: |
| == Definition == | == Vectors == |
| Line 13: | Line 15: |
| Take the multiple as ''x'', so that ''p = ax''. The error term can be characterized as ''b-ax''. | The factor which converts ''a'' into an estimate is notated as ''x̂'', so that ''p = ax̂''. The error term can be characterized by ''e = b - p'' or ''e = b - ax̂''. |
| Line 15: | Line 17: |
| ''a'' is [[LinearAlgebra/Orthogonality|orthogonal]] to ''e''. Therefore, ''a^T^(b-ax) = 0''. This simplifies to ''x = (a^T^b)/(a^T^a)''. Altogether, the projection is characterized as ''p = a(a^T^b)/(a^T^a)''. | ''a'' is [[LinearAlgebra/Orthogonality|orthogonal]] to ''e''. Therefore, ''a^T^(b - ax̂) = 0''. This simplifies to ''x̂ = (a^T^b)/(a^T^a)''. Altogether, the projection is characterized as ''p = a(a^T^b)/(a^T^a)''. |
| Line 17: | Line 19: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is ''(aa^T^)/(a^T^a)''. | A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. Incidentally, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). ---- == Matrices == For systems of equations like '''''A'''x = b'' where there is no solution for ''x'', as in b does not exist in the column space of '''''A''''', we can instead solve '''''A'''x̂ = p'' where ''p'' estimates ''b'' with an error term ''e''. The error term can be characterized as ''e = b - p'' or ''e = b - '''A'''x̂'' ''e'' is [[LinearAlgebra/Orthogonality|orthogonal]] to the row space of '''''A''''' because the error term does not exist in any linear combination of the rows. The projection is more easily worked with in terms of '''''A'''^T^'', so instead think of ''e'' being orthogonal to the column space of '''''A'''^T^'', a.k.a. ''e'' is the [[LinearAlgebra/NullSpaces|null space]] of '''''A'''^T^''. Therefore, '''''A'''^T^(b - '''A'''x̂) = 0''. Altogether, the system of '''normal equations''' for this problem is '''''A'''^T^'''A'''x̂ = '''A'''^T^b''. This simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. Altogether, the projection is characterized as ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is '''''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. ''b'' can also be projected onto ''e'', which geometrically means projecting into the null space of '''''A'''^T^''. Algebraically, if one projection matrix has been computed as '''''P''''', then the projection matrix for going the other way is ''('''I''' - '''P''')b''. As above, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). [[Econometrics/OrdinaryLeastSquares|This should look familiar.]] A projection is inherently the minimization of the error term. Some notes: 1. If '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. 2. If ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. 3. If ''b'' were orthogonal to the column space of '''''A''''', then necessarily ''b'' is in the null space of '''''A'''^T^''. For that reason, projecting ''b'' onto ''e'' would give an identity matrix. In that case, '''''P'''b = 0'' and ''b = e''. |
Projections
When a vector does not exist in s column space, the best approximation of it in that columns space is called a projection.
Contents
Vectors
Given two vectors a and b, we can project b onto a to get the best possible estimate of the former as a multiple of the latter. This projection p has an error term e.
The factor which converts a into an estimate is notated as x̂, so that p = ax̂. The error term can be characterized by e = b - p or e = b - ax̂.
a is orthogonal to e. Therefore, aT(b - ax̂) = 0. This simplifies to x̂ = (aTb)/(aTa). Altogether, the projection is characterized as p = a(aTb)/(aTa).
A matrix P can be defined such that p = Pb. The projection matrix is (aaT)/(aTa). The column space of P (a.k.a. C(P)) is the line through a, and its rank is 1.
Incidentally, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Matrices
For systems of equations like Ax = b where there is no solution for x, as in b does not exist in the column space of A, we can instead solve Ax̂ = p where p estimates b with an error term e.
The error term can be characterized as e = b - p or e = b - Ax̂
e is orthogonal to the row space of A because the error term does not exist in any linear combination of the rows. The projection is more easily worked with in terms of AT, so instead think of e being orthogonal to the column space of AT, a.k.a. e is the null space of AT. Therefore, AT(b - Ax̂) = 0.
Altogether, the system of normal equations for this problem is ATAx̂ = ATb. This simplifies to x̂ = (ATA)-1ATb. Altogether, the projection is characterized as p = A(ATA)-1ATb.
A matrix P can be defined such that p = Pb. The projection matrix is A(ATA)-1AT.
b can also be projected onto e, which geometrically means projecting into the null space of AT. Algebraically, if one projection matrix has been computed as P, then the projection matrix for going the other way is (I - P)b.
As above, P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
This should look familiar. A projection is inherently the minimization of the error term.
Some notes:
If A were a square matrix, most of the above equations would cancel out. But we cannot make that assumption.
If b were in the column space of A, then P would be the identity matrix.
If b were orthogonal to the column space of A, then necessarily b is in the null space of AT. For that reason, projecting b onto e would give an identity matrix. In that case, Pb = 0 and b = e.