Singular Values
Singular values are a generalization of eigenvalues.
Contents
Introduction
It is known that (for square matrices) there are certain vectors which the transformation only scales: Ax = λx.
Similarly, for any matrix A of size m x n, there are certain vectors ui in the column space that correspond to certain vectors vi in the row space. They correspond such that Av = σu. And by composing these vectors and values together: AV = UΣ.
The size of Σ will always match A; that is, it will also be size m x n. V spans the column space of A, so it will be size n x n. U spans the row space, so it will be m x m.
Description
Singular values are square roots of eigenvalues. Specifically, a matrix A has singular values equal to the square roots of the eigenvalues of ATA: σ = √λ.
Matrices of all sizes have singular values. The number of non-zero singular values is equal to the rank of A.
Singular values are also more stable as compared to eigenvalues. Consider the matrix:
┌ ┐ │ 0 1 0 0│ │ 0 0 2 0│ │ 0 0 0 3│ │ 0 0 0 0│ └ ┘
All eigenvalues are 0 and there is only one eigenvector: [1 0 0 0]. By comparison, the singular values are 3, 2, 1, and 0. If the 0 in row 4, column 1 were changed to an extremely small degree, then 4 unique eigenpairs would suddenly emerge. The first three singular values would be unchanged however, and the fourth would become non-zero.