Eigenvalues and Eigenvectors

Matrices are characterized by their eigenvalues and eigenvectors.

Introduction

The square matrix A is a mapping of vectors. There are certain input vectors that map to a linear combination of themself, i.e. A: a -> λa.

Another way to think of this is that there are certain input vectors which maintain their direction through a transformation. For vectors in these directions, the transformation only involves some stretching factor λ. This intuition should also make clear that there are infinitely many such vectors.

These certain input vector are called eigenvectors. Each eigenvector has a corresponding scaling factor λ, called an eigenvalue. They form eigenpairs.

Description

Eigenvalues and eigenvectors are the pairs of λ and x that satisfy Ax = λx and |A - λI| = 0.

Only square matrices have eigenvectors. Given a matrix of size n x n, either there are n unique eigenpairs, or the matrix is defective.

Independent eigenvectors form an eigenbasis, usually notated as S.

The set of eigenvalues of a matrix forms its spectrum.

Properties

Adding nI to A does not change its eigenvectors and adds n to the eigenvalues.

The trace is the sum of eigenvalues. The determinant is the product the eigenvalues.

In an upper or lower triangular matrix, the numbers in the diagonal are its eigenvalues. Furthermore, in a diagonal matrix, the columns are its eigenvectors. This also means that any diagonalizable matrix of size n x n always has n unique eigenpairs, and is never defective.

If a matrix has an eigenvalue of 0, it is singular and non-invertible.

Complex Eigenpairs

Eigenvectors and eigenvalues often include complex numbers.

julia> using LinearAlgebra

julia> A = [0 0 1; 0 1 0; -1 0 0]
3×3 Matrix{Int64}:
  0  0  1
  0  1  0
 -1  0  0

julia> eigvals(A)
3-element Vector{ComplexF64}:
 0.0 - 1.0im
 0.0 + 1.0im
 1.0 + 0.0im

julia> eigvecs(A)
3×3 Matrix{ComplexF64}:
 0.707107-0.0im       0.707107+0.0im       0.0+0.0im
      0.0-0.0im            0.0+0.0im       1.0+0.0im
      0.0-0.707107im       0.0+0.707107im  0.0+0.0im

When a matrix has a complex eigenpair, its conjugate will always be another eigenpair.

Rotation and Eigenvectors

Consider the 2-dimensional rotation matrix:

In two dimensions, there clearly cannot be any vectors which do not change direction through rotation. There are eigenvectors and eigenvalues, but they are complex.

Consider now the 3-dimensional rotation matrix for rotation around the Z axis. This is fundamentally the same rotation.

While 2 of the eigenpairs are complex again, 1 is not. All vectors along the axis of rotation itself are unchanged by this transformation.

A complex eigenpair means that the transformation involves rotation.

Multiplicity

Consider a spectrum of {1, 1, 1, 2, 2, 3}. The eigenvalue 1 is therefore said to have an algebraic multiplicity of 3.

Even if there is a repeated eigenvalue, there may be multiple independent eigenvectors corresponding to it, forming unique eigenpairs. Consider:

julia> using LinearAlgebra

julia> eigen([1 0 0; 0 1 0; 0 0 2])
Eigen{Float64, Float64, Matrix{Float64}, Vector{Float64}}
values:
3-element Vector{Float64}:
 1.0
 1.0
 2.0
vectors:
3×3 Matrix{Float64}:
 1.0  0.0  0.0
 0.0  1.0  0.0
 0.0  0.0  1.0

There are two unique eigenvalues. Despite 1 having algebraic multiplicity of 2, there are 2 unique eigenvectors corresponding to it. Therefore it has geometric multiplicity of 2. (2 has algebraic and geometric multiplicity of 1, incidentally.)

Consider instead:

Clearly this upper triangular matrix has only 1 unique eigenvalue: 1 with algebraic multiplicity of 2. There is only one independent eigenvector associated with that eigenvalue however, so it only has geometric multiplicity of 1.

For each unique eigenvalue, the geometric multiplicity is always equal to or less than algebraic multiplicity. The algebraic multiplicity is the dimensionality of the eigenspace, while the geometric multiplicity is the dimensions spanned by the eigenbasis.

To reiterate, a diagonalizable matrix of size n x n always has n unique eigenpairs. This follows from the fact that a diagonalizable matrix must have algebraic multiplicity equal to geometric multiplicity for all eigenvalues.

Solving for Eigenvalues

Trivial Case

If a matrix is upper triangular, lower triangular, or diagonal, then the numbers on the diagonal are the eigenvalues.

Simple Case

The eigenvalues of a 2 x 2 matrix can sometimes be identified through the determinant and trace. Recall that:

• the trace is the sum of eigenvalues
• the determinant is the product of eigenvalues

A little bit of mental math can usually solve the system given by |A| = λ₁ * λ₂ and tr(A) = λ₁ + λ₂.

Conventional Method

Because eigenvalues are characterized by |A - λI| = 0, they can be solved for by:

• subtracting λ from each value on the diagonal

• formulating the determinant for this difference
• setting the formulation for 0

• solving for λ

In a simple 2 x 2 matrix, this looks like:

|    A   -  λI   | = 0

│ ┌    ┐  ┌    ┐ │
│ │ a b│ -│ λ 0│ │ = 0
│ │ c d│  │ 0 λ│ │
│ └    ┘  └    ┘ │

│ ┌        ┐ │
│ │ a-λ   b│ │ = 0
│ │   c d-λ│ │
│ └        ┘ │

(a-λ)(d-λ) - bc = 0

This leads to the characteristic polynomial of A; solving for the roots, as through either factorization or the quadratic formula, gives the eigenvalues.

Algebraic Method

Recall again that:

• the trace is the sum of eigenvalues
• the determinant is the product of eigenvalues

It follows that, for a 2 x 2 matrix, 1/2 of the trace is equal to the mean of the eigenvalues.

Furthermore, because the characteristic polynomial of a R² matrix is quadratic, the eigenvalues must be evenly spaced from the center (i.e., the mean). The mean (m) is known from the above properties. The eigenvectors can be expressed as m-d and m+d.

Recall once more that the determinant is the product of eigenvalues. It must be that |A| = (m-d)(m+d) = m² - d².

Altogether,

Solving for Eigenvectors

Trivial Cases

If a matrix is diagonal, then the columns are the eigenvectors.

If one of the eigenvectors of a 2 x 2 matrix is known and it is complex, then it conjugate must be the other eigenvector.

Conventional Method

This assumes that the eigenvalues have already been solved for.

Eigenvectors are solved as the null space of A - λI. Which is to say, for a given λ, solve for x in the system (A - λI)x = 0.

In a simple 2 x 2 matrix, this looks like:

(   A    -  λI   )   x  =   0

/ ┌    ┐  ┌    ┐ \ ┌  ┐   ┌  ┐
│ │ a b│ -│ λ 0│ │ │ u│ = │ 0│
│ │ c d│  │ 0 λ│ │ │ v│   │ 0│
\ └    ┘  └    ┘ / └  ┘   └  ┘

┌        ┐ ┌  ┐   ┌  ┐
│ a-λ   b│ │ u│ = │ 0│
│   c d-λ│ │ v│   │ 0│
└        ┘ └  ┘   └  ┘

┌        ┐ ┌  ┐   ┌  ┐
│ a-λ   b│ │ u│ = │ 0│
│   c d-λ│ │ v│   │ 0│
└        ┘ └  ┘   └  ┘

(a-λ)u + (b)v = 0
(c)u + (d-λ)v = 0

Then proceed with any method for solving for the null space. As an example:

julia> using LinearAlgebra

julia> A = [2 1; 1 2]
2×2 Matrix{Int64}:
 2  1
 1  2

julia> eigvals, eigvecs = eigen(A)
Eigen{Float64, Float64, Matrix{Float64}, Vector{Float64}}
values:
2-element Vector{Float64}:
 1.0
 3.0
vectors:
2×2 Matrix{Float64}:
 -0.707107  0.707107
  0.707107  0.707107

julia> nullspace(A - eigvals[1]*I)
2×1 Matrix{Float64}:
 -0.7071067811865475
  0.7071067811865476

CategoryRicottone