|
Size: 729
Comment: Initial commit
|
Size: 3077
Comment: Rewrite
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
When a vector does not exist in a column space, the '''projection''' is the best approximation of it in linear combinations of that column space. |
|
| Line 9: | Line 11: |
| == Definition == | == Vectors == |
| Line 11: | Line 13: |
| Given two vectors ''a'' and ''b'', we can '''project''' ''b'' onto ''a'' to get the best possible estimate of the former as a multiple of the latter. This projection ''p'' has an error term ''e''. | Given vectors ''a'' and ''b'', ''b'' can be projected into ''C(a)'', the column space of ''a''. This projection ''p'' has an error term ''e''. |
| Line 13: | Line 15: |
| Take the multiple as ''x'', so that ''p = ax''. The error term can be characterized as ''b-ax''. | The factor which converts ''a'' into an estimate is notated as ''x̂'', so that ''p = ax̂''. The '''error term''' can be characterized by ''e = b - p'' or ''e = b - ax̂''. ''a'' is [[LinearAlgebra/Orthogonality|orthogonal]] to ''e'', so ''a^T^(b - ax̂) = 0''. This simplifies to ''x̂ = (a^T^b)/(a^T^a)''. Altogether, the projection is ''p = a(a^T^b)/(a^T^a)''. |
| Line 15: | Line 17: |
| ''a'' is [[LinearAlgebra/Orthogonality|orthogonal]] to ''e''. Therefore, ''a^T^(b-ax) = 0''. This simplifies to ''x = (a^T^b)/(a^T^a)''. Altogether, the projection is characterized as ''p = a(a^T^b)/(a^T^a)''. | The '''projection matrix''' '''''P''''' satisfies ''p = '''P'''b''. It is calculated ''(aa^T^)/(a^T^a)''. ''C('''P''')'', the column space of '''''P''''', is equivalent to the column space of ''a''. (It follows that '''''P''''' is also of [[LinearAlgebra/Rank|rank]] 1.) |
| Line 17: | Line 19: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The projection matrix is ''(aa^T^)/(a^T^a)''. | === Properties === The projection matrix '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotency|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). ---- == Matrices == Given a system as '''''A'''x = b'', if ''b'' is not in ''C('''A''')'', the column space of '''''A''''', then there is no possible solution for ''x''. The best approximation is expressed as '''''A'''x̂ = p'' where projection ''p'' estimates ''b'' with an error term ''e''. The error term can be characterized by ''e = b - p'' or ''e = b - '''A'''x̂''. ''e'' is orthogonal to ''R('''A''')'', the row space of '''''A'''''; equivalently it is orthogonal to ''C('''A'''^T^)''. Orthogonality in this context means that ''e'' is in the [[LinearAlgebra/NullSpaces|null space]], so '''''A'''^T^(b - '''A'''x̂) = 0''. The system of '''normal equations''' is '''''A'''^T^'''A'''x̂ = '''A'''^T^b''. This simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. Altogether, the projection is characterized by ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. The projection matrix '''''P''''' satisfies ''p = '''P'''b''. It is calculated as '''''P''' = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. ''b'' can also be projected onto ''e'', which geometrically means projecting into the null space of '''''A'''^T^''. Algebraically, if one projection matrix has been computed as '''''P''''', then the projection matrix for going the other way is ''('''I''' - '''P''')b''. === Properties === As above, the projection matrix '''''P''''' is symmetric and idempotent. If '''''A''''' is square, the above equations simplify rapidly. If ''b'' actually ''was'' in ''C('''A''')'', then '''''P''' = '''I'''''. Conversely, if ''b'' is orthogonal to ''C('''A''')'', then '''''P'''b = 0'' and ''b = e''. === Usage === [[Statistics/OrdinaryLeastSquares|This should look familiar.]] A projection is inherently the minimization of the error term. |
Projections
When a vector does not exist in a column space, the projection is the best approximation of it in linear combinations of that column space.
Contents
Vectors
Given vectors a and b, b can be projected into C(a), the column space of a. This projection p has an error term e.
The factor which converts a into an estimate is notated as x̂, so that p = ax̂. The error term can be characterized by e = b - p or e = b - ax̂. a is orthogonal to e, so aT(b - ax̂) = 0. This simplifies to x̂ = (aTb)/(aTa). Altogether, the projection is p = a(aTb)/(aTa).
The projection matrix P satisfies p = Pb. It is calculated (aaT)/(aTa). C(P), the column space of P, is equivalent to the column space of a. (It follows that P is also of rank 1.)
Properties
The projection matrix P is symmetric (i.e. PT = P) and idempotent (i.e. P2 = P).
Matrices
Given a system as Ax = b, if b is not in C(A), the column space of A, then there is no possible solution for x. The best approximation is expressed as Ax̂ = p where projection p estimates b with an error term e.
The error term can be characterized by e = b - p or e = b - Ax̂. e is orthogonal to R(A), the row space of A; equivalently it is orthogonal to C(AT). Orthogonality in this context means that e is in the null space, so AT(b - Ax̂) = 0.
The system of normal equations is ATAx̂ = ATb. This simplifies to x̂ = (ATA)-1ATb. Altogether, the projection is characterized by p = A(ATA)-1ATb.
The projection matrix P satisfies p = Pb. It is calculated as P = A(ATA)-1AT.
b can also be projected onto e, which geometrically means projecting into the null space of AT. Algebraically, if one projection matrix has been computed as P, then the projection matrix for going the other way is (I - P)b.
Properties
As above, the projection matrix P is symmetric and idempotent.
If A is square, the above equations simplify rapidly.
If b actually was in C(A), then P = I. Conversely, if b is orthogonal to C(A), then Pb = 0 and b = e.
Usage
This should look familiar. A projection is inherently the minimization of the error term.