|
Size: 3625
Comment: Made consistent
|
← Revision 20 as of 2026-02-16 16:43:48 ⇥
Size: 2753
Comment: Fixed link
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Projections = | = Projection = |
| Line 3: | Line 3: |
| When two vectors do not exist in the same column space, the best approximation of one in the other's columns space is called a '''projection'''. | A '''projection''' is an approximation within a column space. |
| Line 11: | Line 11: |
| Line 13: | Line 14: |
| Given two vectors ''a'' and ''b'', we can '''project''' ''b'' onto ''a'' to get the best possible estimate of the former as a multiple of the latter. This projection ''p'' has an error term ''e''. | Given two vectors ''a'' and ''b'', ''b'' can be [[Calculus/Projection|projected]] into ''C(a)'', the column space of ''a''. |
| Line 15: | Line 16: |
| The factor which converts ''a'' into an estimate is notated as ''x̂'', so that ''p = ax̂''. The error term can be characterized by ''e = b - p'' or ''e = b - ax̂''. | Furthermore, the projection vector with the least [[Calculus/Distance#Euclidean_distance|error]] as compared to the true vector ''b'' is characterized by orthogonality. Let ''e'' be the error vector; it is orthogonal to ''a''. |
| Line 17: | Line 18: |
| ''a'' is [[LinearAlgebra/Orthogonality|orthogonal]] to ''e''. Therefore, ''a^T^(b - ax̂) = 0''. This simplifies to ''x̂ = (a^T^b)/(a^T^a)''. Altogether, the projection is characterized as ''p = a(a^T^b)/(a^T^a)''. | The [[LinearAlgebra/LinearMapping|transformation]] of vector ''b'' into projection vector ''p'' can be described by a '''projection matrix'''. It is notated '''''P''''' as in ''p = '''P'''b''. |
| Line 19: | Line 20: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is ''(aa^T^)/(a^T^a)''. The column space of '''''P''''' (a.k.a. ''C('''P''')'') is the line through ''a'', and its rank is 1. | |
| Line 21: | Line 21: |
| Incidentally, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). | === Properties === The projection matrix '''''P''''' satisfying ''p = '''P'''b'' is [[LinearAlgebra/Rank|rank]] 1. ''C('''P''')'', the column space of the projection matrix, is equivalent to ''C(b)''. Projection matrices are [[LinearAlgebra/SpecialMatrices#Symmetric_Matrices|symmetric]] (i.e., '''''P'''^T^ = '''P''''') and [[LinearAlgebra/Idempotency|idempotent]] (i.e., '''''P'''^2^ = '''P'''''). |
| Line 29: | Line 36: |
| For systems of equations like '''''A'''x = b'' where there is no solution for ''x'', as in b does not exist in the column space of '''''A''''', we can instead solve '''''A'''x̂ = p'' where ''p'' estimates ''b'' with an error term ''e''. | For all the same reasons, a vector ''b'' can be projected into ''C('''A''')'', the column space of '''''A'''''. The [[Calculus/Distance#Euclidean_distance|error]] vector ''e'' is orthogonal to ''R('''A''')'', the row space of '''''A''''', and is therefore in the [[LinearAlgebra/NullSpace|null space]]. |
| Line 31: | Line 38: |
| The error term can be characterized as ''e = b - p'' or ''e = b - '''A'''x̂'' | The projection matrix is now notated '''''P''''' as in ''p = '''P'''b''. |
| Line 33: | Line 40: |
| ''e'' is [[LinearAlgebra/Orthogonality|orthogonal]] to the row space of '''''A''''' because the error term does not exist in any linear combination of the rows. The projection is more easily worked with in terms of '''''A'''^T^'', so instead think of ''e'' being orthogonal to the column space of '''''A'''^T^'', a.k.a. ''e'' is the [[LinearAlgebra/NullSpaces|null space]] of '''''A'''^T^''. Therefore, '''''A'''^T^(b - '''A'''x̂) = 0''. | |
| Line 35: | Line 41: |
| Altogether, the system of '''normal equations''' for this problem is '''''A'''^T^'''A'''x̂ = '''A'''^T^b''. This simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. Altogether, the projection is characterized as ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. | |
| Line 37: | Line 42: |
| A matrix '''''P''''' can be defined such that ''p = '''P'''b''. The '''projection matrix''' is '''''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. | === Least Squares === |
| Line 39: | Line 44: |
| ''b'' can also be projected onto ''e'', which geometrically means projecting into the null space of '''''A'''^T^''. Algebraically, if one projection matrix has been computed as '''''P''''', then the projection matrix for going the other way is ''('''I''' - '''P''')b''. | Given a consistent system as '''''A'''x = b'', i.e. ''b'' is in ''C('''A''')'', there are solutions for ''x''. |
| Line 41: | Line 46: |
| As above, '''''P''''' is [[LinearAlgebra/MatrixProperties#Symmetry|symmetric]] (i.e. '''''P'''^T^ = '''P''''') and [[LinearAlgebra/MatrixProperties#Idempotent|idempotent]] (i.e. '''''P'''^2^ = '''P'''''). | If the system is inconsistent, then there is no solution. The best approximation is expressed as '''''A'''x̂ = p'' where projection ''p'' estimates ''b'' with an error term ''e''. [[Statistics/OrdinaryLeastSquares|This should sound familiar.]] |
| Line 43: | Line 48: |
| [[Econometrics/OrdinaryLeastSquares|This should look familiar.]] A projection is inherently the minimization of the error term. | The error term can be generally characterized by ''e = b - p''. An expression for ''p'' is known, so ''e = b - '''A'''x̂''. |
| Line 45: | Line 50: |
| Some notes: | ''e'' is orthogonal to ''R('''A''')'', so '''''A'''^T^e = 0''. Substituting in the above expression gives '''''A'''^T^(b - '''A'''x̂) = 0'' |
| Line 47: | Line 52: |
| 1. If '''''A''''' were a square matrix, most of the above equations would [[LinearAlgebra/MatrixInversion|cancel out]]. But we cannot make that assumption. 2. If ''b'' were in the column space of '''''A''''', then '''''P''''' would be the identity matrix. 3. If ''b'' were orthogonal to the column space of '''''A''''', then necessarily ''b'' is in the null space of '''''A'''^T^''. For that reason, projecting ''b'' onto ''e'' would give an identity matrix. In that case, '''''P'''b = 0'' and ''b = e''. |
Altogether, '''''A'''^T^'''A'''x̂ = '''A'''^T^b'' which simplifies to ''x̂ = ('''A'''^T^'''A''')^-1^'''A'''^T^b''. The projection matrix is calculated as '''''P''' = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^''. The projection is calculated as ''p = '''A'''('''A'''^T^'''A''')^-1^'''A'''^T^b''. === Properties === Projection matrices are still symmetric and idempotent. If ''b'' is in ''C('''A''')'', then '''''P''' = '''I'''''.. Conversely, if ''b'' is orthogonal to ''C('''A''')'', then '''''P'''b = 0'' and ''b = e''. |
Projection
A projection is an approximation within a column space.
Vectors
Given two vectors a and b, b can be projected into C(a), the column space of a.
Furthermore, the projection vector with the least error as compared to the true vector b is characterized by orthogonality. Let e be the error vector; it is orthogonal to a.
The transformation of vector b into projection vector p can be described by a projection matrix. It is notated P as in p = Pb.
Properties
The projection matrix P satisfying p = Pb is rank 1.
C(P), the column space of the projection matrix, is equivalent to C(b).
Projection matrices are symmetric (i.e., PT = P) and idempotent (i.e., P2 = P).
Matrices
For all the same reasons, a vector b can be projected into C(A), the column space of A. The error vector e is orthogonal to R(A), the row space of A, and is therefore in the null space.
The projection matrix is now notated P as in p = Pb.
Least Squares
Given a consistent system as Ax = b, i.e. b is in C(A), there are solutions for x.
If the system is inconsistent, then there is no solution. The best approximation is expressed as Ax̂ = p where projection p estimates b with an error term e. This should sound familiar.
The error term can be generally characterized by e = b - p. An expression for p is known, so e = b - Ax̂.
e is orthogonal to R(A), so ATe = 0. Substituting in the above expression gives AT(b - Ax̂) = 0
Altogether, ATAx̂ = ATb which simplifies to x̂ = (ATA)-1ATb.
The projection matrix is calculated as P = A(ATA)-1AT.
The projection is calculated as p = A(ATA)-1ATb.
Properties
Projection matrices are still symmetric and idempotent.
If b is in C(A), then P = I.. Conversely, if b is orthogonal to C(A), then Pb = 0 and b = e.