Orthogonality

Orthogonality is an important property for relating two vectors, or two subspaces, or a vector and a plane. The math notation is ⊥, as in x ⊥ y.

Orthonormality is an expanded concept, requiring that the components be unit vectors.

Vectors

The concept of orthogonality is a generalization of perpendicularity from 2-dimensional space.

Put simply, vectors a and b are proven to be orthogonal if their dot product is 0.

More precisely: assuming orthogonality, vectors a and b will satisfy the Pythagorean theorem. The hypotenuse is Euclidean distance: (a+b)^T(a+b). Simplifying from there:

a^Ta + b^Tb = (a+b)^T(a+b)

a^Ta + b^Tb = a^Ta + b^Tb + a^Tb + b^Ta

0 = a^Tb + b^Ta

0 = 2(a^Tb)

0 = a^Tb

Subspaces

For a subspace S to be orthogonal to a subspace T, every vector in S must be orthogonal to every vector in T.

Vectors and Planes

The null space of a matrix A contains the vectors that are not in the row space. These vectors cancel out; they are not a linear combination of the rows; if the row space is a plane, then these vectors are not on that plane.

The null space of A (a.k.a. N(A)) is orthogonal to the row space of A (a.k.a. R(A)). The null space of A^T (a.k.a. N(A^T)) is orthogonal to the column space of A (a.k.a. C(A)). Commonly this means that the row and column spaces are planes while the null spaces of A and A^T are vectors, but that isn't always true.

Matrices

If a matrix is composed of orthonormal columns, then it is a matrix with orthonormal columns. These are usually denoted as Q. This has an important property: Q^TQ = I.

The projection of A if A is a matrix with orthonormal columns simplifies from P = A(A^TA^-1)A^T into P = QQ^T. Correspondingly, the system of normal equations simplifies from A^TAx̂ = A^Tb into x̂ = Q^Tb.

If a matrix with orthonormal columns is also square, only then can it be called an orthogonal matrix. This has an additional important property: Q^T = Q^-1.

For example, if Q is square, then the projection matrix further simplifies to P = I.

CategoryRicottone