|
Size: 3611
Comment: Typo
|
Size: 2861
Comment: Simplifying matrix page names
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Projections = | = Projection = |
| Line 3: | Line 3: |
| When a vector does not exist in s column space, the best approximation of it in that columns space is called a '''projection'''. | A '''projection''' is an approximation within a column space. A '''projection matrix''' describes the linear transformation of the approximation. See also [[Calculus/Projection|vector projection]]. |
| Line 11: | Line 13: |
| Line 13: | Line 16: |
| Given two vectors ''a'' and ''b'', we can '''project''' ''b'' onto ''a'' to get the best possible estimate of the former as a multiple of the latter. This projection ''p'' has an error term ''e''. | Given vectors ''a'' and ''b'', ''b'' can be [[Calculus/Projection|projected]] into the column space of ''a'' (i.e., ''C(a)''). A '''projection matrix''' describes the linear transformation from vector ''a'' to projection vector ''p''. Such a projection matrix '''''P''''' satisfies ''p = '''P'''a''. |
| Line 15: | Line 18: |
| The factor which converts ''a'' into an estimate is notated as ''x̂'', so that ''p = ax̂''. The error term can be characterized by ''e = b - p'' or ''e = b - ax̂''. | The column space of the projection matrix (i.e., ''C('''P''')'') is equivalent to ''C(b)''. It follows that '''''P''''' is also of [[LinearAlgebra/Rank|rank]] 1. |
| Line 17: | Line 20: |
| ''a'' is [[LinearAlgebra/Orthogonality|orthogonal]] to ''e''. Therefore, ''a^T^(b - ax̂) = 0''. This simplifies to ''x̂ = (a^T^b)/(a^T^a)''. Altogether, the projection is characterized as ''p = a(a^T^b)/(a^T^a)''. | |
| Line 19: | Line 21: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. | |
| Line 21: | Line 22: |
| Incidentally, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotency|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). | === Properties === Projection matrices are [[LinearAlgebra/SpecialMatrices#Symmetric_Matrices|symmetric]] (i.e., '''''P'''^T^ = '''P''''') and [[LinearAlgebra/Idempotency|idempotent]] (i.e., '''''P'''^2^ = '''P'''''). |
| Line 29: | Line 32: |
| For systems of equations like '''''A'''x = b'' where there is no solution for ''x'', as in b does not exist in the column space of '''''A''''', we can instead solve '''''A'''x̂ = p'' where ''p'' estimates ''b'' with an error term ''e''. | Given vector ''b'' and matrix '''''A''''', ''b'' can be projected into the column space of '''''A''''' (i.e., ''C('''A''')''). |
| Line 31: | Line 34: |
| The error term can be characterized as ''e = b - p'' or ''e = b - '''A'''x̂'' | Given a linear system as '''''A'''x = b'', if ''b'' is in ''C('''A''')'', there are solutions for ''x''. If ''b'' is ''not'' in ''C('''A''')'' however, there is no possible solution. The best ''approximation'' is expressed as '''''A'''x̂ = p'' where projection ''p'' estimates ''b'' with an error term ''e''. [[Statistics/OrdinaryLeastSquares|This should sound familiar.]] |
| Line 33: | Line 36: |
| ''e'' is [[LinearAlgebra/Orthogonality|orthogonal]] to the row space of '''''A''''' because the error term does not exist in any linear combination of the rows. The projection is more easily worked with in terms of '''''A'''^T^'', so instead think of ''e'' being orthogonal to the column space of '''''A'''^T^'', a.k.a. ''e'' is the [[LinearAlgebra/NullSpaces|null space]] of '''''A'''^T^''. Therefore, '''''A'''^T^(b - '''A'''x̂) = 0''. | The error term can be characterized by ''e = b - p'' or ''e = b - '''A'''x̂''. ''e'' is orthogonal to ''R('''A''')'', the row space of '''''A'''''; equivalently it is orthogonal to ''C('''A'''^T^)''. Orthogonality in this context means that ''e'' is in the [[LinearAlgebra/NullSpace|null space]], so '''''A'''^T^(b - '''A'''x̂) = 0''. |
| Line 35: | Line 38: |
| Altogether, the system of '''normal equations''' for this problem is '''''A'''^T^'''A'''x̂ = '''A'''^T^b''. This simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. Altogether, the projection is characterized as ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. | The system of '''normal equations''' is '''''A'''^T^'''A'''x̂ = '''A'''^T^b''. This simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. Altogether, the projection is characterized by ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. |
| Line 37: | Line 40: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is '''''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. | The projection matrix '''''P''''' satisfies ''p = '''P'''b''. It is calculated as '''''P''' = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. |
| Line 41: | Line 44: |
| As above, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotency|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). | |
| Line 43: | Line 45: |
| [[Econometrics/OrdinaryLeastSquares|This should look familiar.]] A projection is inherently the minimization of the error term. | |
| Line 45: | Line 46: |
| Some notes: | === Properties === |
| Line 47: | Line 48: |
| 1. If '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. 2. If ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. 3. If ''b'' were orthogonal to the column space of '''''A''''', then necessarily ''b'' is in the null space of '''''A'''^T^''. For that reason, projecting ''b'' onto ''e'' would give an identity matrix. In that case, '''''P'''b = 0'' and ''b = e''. |
As above, the projection matrix '''''P''''' is symmetric and idempotent. If '''''A''''' is square, the above equations simplify rapidly. If ''b'' is in ''C('''A''')'', then '''''P''' = '''I'''''. Conversely, if ''b'' is orthogonal to ''C('''A''')'', then '''''P'''b = 0'' and ''b = e''. |
Projection
A projection is an approximation within a column space. A projection matrix describes the linear transformation of the approximation.
See also vector projection.
Contents
Vectors
Given vectors a and b, b can be projected into the column space of a (i.e., C(a)). A projection matrix describes the linear transformation from vector a to projection vector p. Such a projection matrix P satisfies p = Pa.
The column space of the projection matrix (i.e., C(P)) is equivalent to C(b). It follows that P is also of rank 1.
Properties
Projection matrices are symmetric (i.e., PT = P) and idempotent (i.e., P2 = P).
Matrices
Given vector b and matrix A, b can be projected into the column space of A (i.e., C(A)).
Given a linear system as Ax = b, if b is in C(A), there are solutions for x. If b is not in C(A) however, there is no possible solution. The best approximation is expressed as Ax̂ = p where projection p estimates b with an error term e. This should sound familiar.
The error term can be characterized by e = b - p or e = b - Ax̂. e is orthogonal to R(A), the row space of A; equivalently it is orthogonal to C(AT). Orthogonality in this context means that e is in the null space, so AT(b - Ax̂) = 0.
The system of normal equations is ATAx̂ = ATb. This simplifies to x̂ = (ATA)-1ATb. Altogether, the projection is characterized by p = A(ATA)-1ATb.
The projection matrix P satisfies p = Pb. It is calculated as P = A(ATA)-1AT.
b can also be projected onto e, which geometrically means projecting into the null space of AT. Algebraically, if one projection matrix has been computed as P, then the projection matrix for going the other way is (I - P)b.
Properties
As above, the projection matrix P is symmetric and idempotent.
If A is square, the above equations simplify rapidly.
If b is in C(A), then P = I. Conversely, if b is orthogonal to C(A), then Pb = 0 and b = e.