|
Size: 3077
Comment: Rewrite
|
← Revision 20 as of 2026-02-16 16:43:48 ⇥
Size: 2753
Comment: Fixed link
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Projections = | = Projection = |
| Line 3: | Line 3: |
| When a vector does not exist in a column space, the '''projection''' is the best approximation of it in linear combinations of that column space. | A '''projection''' is an approximation within a column space. |
| Line 11: | Line 11: |
| Line 13: | Line 14: |
| Given vectors ''a'' and ''b'', ''b'' can be projected into ''C(a)'', the column space of ''a''. This projection ''p'' has an error term ''e''. | Given two vectors ''a'' and ''b'', ''b'' can be [[Calculus/Projection|projected]] into ''C(a)'', the column space of ''a''. |
| Line 15: | Line 16: |
| The factor which converts ''a'' into an estimate is notated as ''x̂'', so that ''p = ax̂''. The '''error term''' can be characterized by ''e = b - p'' or ''e = b - ax̂''. ''a'' is [[LinearAlgebra/Orthogonality|orthogonal]] to ''e'', so ''a^T^(b - ax̂) = 0''. This simplifies to ''x̂ = (a^T^b)/(a^T^a)''. Altogether, the projection is ''p = a(a^T^b)/(a^T^a)''. | Furthermore, the projection vector with the least [[Calculus/Distance#Euclidean_distance|error]] as compared to the true vector ''b'' is characterized by orthogonality. Let ''e'' be the error vector; it is orthogonal to ''a''. |
| Line 17: | Line 18: |
| The '''projection matrix''' '''''P''''' satisfies ''p = '''P'''b''. It is calculated ''(aa^T^)/(a^T^a)''. ''C('''P''')'', the column space of '''''P''''', is equivalent to the column space of ''a''. (It follows that '''''P''''' is also of [[LinearAlgebra/Rank|rank]] 1.) | The [[LinearAlgebra/LinearMapping|transformation]] of vector ''b'' into projection vector ''p'' can be described by a '''projection matrix'''. It is notated '''''P''''' as in ''p = '''P'''b''. |
| Line 23: | Line 24: |
| The projection matrix '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotency|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). | The projection matrix '''''P''''' satisfying ''p = '''P'''b'' is [[LinearAlgebra/Rank|rank]] 1. ''C('''P''')'', the column space of the projection matrix, is equivalent to ''C(b)''. Projection matrices are [[LinearAlgebra/SpecialMatrices#Symmetric_Matrices|symmetric]] (i.e., '''''P'''^T^ = '''P''''') and [[LinearAlgebra/Idempotency|idempotent]] (i.e., '''''P'''^2^ = '''P'''''). |
| Line 31: | Line 36: |
| Given a system as '''''A'''x = b'', if ''b'' is not in ''C('''A''')'', the column space of '''''A''''', then there is no possible solution for ''x''. The best approximation is expressed as '''''A'''x̂ = p'' where projection ''p'' estimates ''b'' with an error term ''e''. | For all the same reasons, a vector ''b'' can be projected into ''C('''A''')'', the column space of '''''A'''''. The [[Calculus/Distance#Euclidean_distance|error]] vector ''e'' is orthogonal to ''R('''A''')'', the row space of '''''A''''', and is therefore in the [[LinearAlgebra/NullSpace|null space]]. |
| Line 33: | Line 38: |
| The error term can be characterized by ''e = b - p'' or ''e = b - '''A'''x̂''. ''e'' is orthogonal to ''R('''A''')'', the row space of '''''A'''''; equivalently it is orthogonal to ''C('''A'''^T^)''. Orthogonality in this context means that ''e'' is in the [[LinearAlgebra/NullSpaces|null space]], so '''''A'''^T^(b - '''A'''x̂) = 0''. | The projection matrix is now notated '''''P''''' as in ''p = '''P'''b''. |
| Line 35: | Line 40: |
| The system of '''normal equations''' is '''''A'''^T^'''A'''x̂ = '''A'''^T^b''. This simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. Altogether, the projection is characterized by ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. | |
| Line 37: | Line 41: |
| The projection matrix '''''P''''' satisfies ''p = '''P'''b''. It is calculated as '''''P''' = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. | |
| Line 39: | Line 42: |
| ''b'' can also be projected onto ''e'', which geometrically means projecting into the null space of '''''A'''^T^''. Algebraically, if one projection matrix has been computed as '''''P''''', then the projection matrix for going the other way is ''('''I''' - '''P''')b''. | === Least Squares === Given a consistent system as '''''A'''x = b'', i.e. ''b'' is in ''C('''A''')'', there are solutions for ''x''. If the system is inconsistent, then there is no solution. The best approximation is expressed as '''''A'''x̂ = p'' where projection ''p'' estimates ''b'' with an error term ''e''. [[Statistics/OrdinaryLeastSquares|This should sound familiar.]] The error term can be generally characterized by ''e = b - p''. An expression for ''p'' is known, so ''e = b - '''A'''x̂''. ''e'' is orthogonal to ''R('''A''')'', so '''''A'''^T^e = 0''. Substituting in the above expression gives '''''A'''^T^(b - '''A'''x̂) = 0'' Altogether, '''''A'''^T^'''A'''x̂ = '''A'''^T^b'' which simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. The projection matrix is calculated as '''''P''' = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. The projection is calculated as ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. |
| Line 45: | Line 62: |
| As above, the projection matrix '''''P''''' is symmetric and idempotent. | Projection matrices are still symmetric and idempotent. |
| Line 47: | Line 64: |
| If '''''A''''' is square, the above equations simplify rapidly. If ''b'' actually ''was'' in ''C('''A''')'', then '''''P''' = '''I'''''. Conversely, if ''b'' is orthogonal to ''C('''A''')'', then '''''P'''b = 0'' and ''b = e''. === Usage === [[Statistics/OrdinaryLeastSquares|This should look familiar.]] A projection is inherently the minimization of the error term. |
If ''b'' is in ''C('''A''')'', then '''''P''' = '''I'''''.. Conversely, if ''b'' is orthogonal to ''C('''A''')'', then '''''P'''b = 0'' and ''b = e''. |
Projection
A projection is an approximation within a column space.
Vectors
Given two vectors a and b, b can be projected into C(a), the column space of a.
Furthermore, the projection vector with the least error as compared to the true vector b is characterized by orthogonality. Let e be the error vector; it is orthogonal to a.
The transformation of vector b into projection vector p can be described by a projection matrix. It is notated P as in p = Pb.
Properties
The projection matrix P satisfying p = Pb is rank 1.
C(P), the column space of the projection matrix, is equivalent to C(b).
Projection matrices are symmetric (i.e., PT = P) and idempotent (i.e., P2 = P).
Matrices
For all the same reasons, a vector b can be projected into C(A), the column space of A. The error vector e is orthogonal to R(A), the row space of A, and is therefore in the null space.
The projection matrix is now notated P as in p = Pb.
Least Squares
Given a consistent system as Ax = b, i.e. b is in C(A), there are solutions for x.
If the system is inconsistent, then there is no solution. The best approximation is expressed as Ax̂ = p where projection p estimates b with an error term e. This should sound familiar.
The error term can be generally characterized by e = b - p. An expression for p is known, so e = b - Ax̂.
e is orthogonal to R(A), so ATe = 0. Substituting in the above expression gives AT(b - Ax̂) = 0
Altogether, ATAx̂ = ATb which simplifies to x̂ = (ATA)-1ATb.
The projection matrix is calculated as P = A(ATA)-1AT.
The projection is calculated as p = A(ATA)-1ATb.
Properties
Projection matrices are still symmetric and idempotent.
If b is in C(A), then P = I.. Conversely, if b is orthogonal to C(A), then Pb = 0 and b = e.