Spectral Theorem

In linear algebra, one is often interested in the canonical forms of a linear transformation. Given a particularly nice basis for the vector spaces in which one is working, the matrix of a linear transformation may also be particularly nice, revealing some information about how the transformation operates on the vector space.

The spectral theorem provides a sufficient criterion for the existence of a particular canonical form. Specifically, the spectral theorem states that if $M$ equals the transpose of $M$, then $M$ is diagonalizable: there exists an invertible matrix $C$ such that $C^{-1} MC$ is a diagonal matrix. Recall that a diagonal matrix is any matrix for which all entries off the main diagonal (the diagonal from top left to bottom right) are zero.

A matrix $M$ with entries in $\mathbb{R}$ is called symmetric if $M =M^{T}$. The spectral theorem states that any symmetric matrix is diagonalizable.

Consider the matrix

$A= \begin{pmatrix} 2 & 6 \\ 0 & -1 \end{pmatrix}.$

The eigenvalues of this matrix are $\lambda_1, \lambda_2 = 2, -1$, with corresponding eigenvectors $v_1 = (1,0)$ and $v_2 = (-2,1)$. These eigenvectors $\{v_1, v_2\}$ form a basis for $\mathbb{R}^2$, with change of basis matrix

$C= \begin{pmatrix} 1 & -2 \\ 0 & 1 \end{pmatrix},$

sending the standard basis vector $e_1 = (1,0)$ to $v_1$ and $e_2 = (0,1)$ to $v_2$. In the basis $\{v_1, v_2\}$, the matrix for the linear transformation described by $A$ is precisely

$C^{-1} AC = \begin{pmatrix} 2 & 0 \\ 0 & -1 \end{pmatrix}.$

This diagonal matrix is easier to parse than $A$; looking at it immediately tells one that $A$ acts on $\mathbb{R}^2$ by scaling two axes, one by a factor of $2$ and one by a factor of $-1$ (i.e. reflection).

$A$ acts on $\mathbb{R}^2$ by scaling along the two axes depicted, the lines $y = 0$ and $y = -\frac x2$.

A matrix $M$ is called diagonalizable if there is a basis in which the linear transformation described by $M$ has a diagonal matrix, i.e. a matrix whose entries off the main diagonal (the diagonal from top left to bottom right) are all zero. Equivalently, $M$ is diagonalizable if and only if there exists an invertible matrix $C$ such that $C^{-1}M C$ is a diagonal matrix. Diagonalizable matrices are easier to work with than non-diagonalizable matrices since they can be placed in this canonical diagonal form with a change of basis.

Let's start off by proving the following useful lemma:

Lemma: All the eigenvalues of a real $n \times n$ symmetric matrix $M$ are real.

Proof:
Let $\lambda \in \mathbb{C}$ be an eigenvalue of $M$ with corresponding eigenvector $\ v \in \mathbb{C^n}$. Now I will show that $\ \overline{\lambda} = \lambda$ by evaluating $\ (Mv)^{T} \overline{v}$ in two ways:

$\begin{aligned} \ (Mv)^{T} \overline{v} &= v^{T}M^{T}\overline{v} \\ &= v^{T}(M\overline{v}) \\ &= v^{T}\big(\overline{M}\overline{v}\big) \\ &= v^{T}\big(\overline{Mv}\big) \\ &= v^{T}\big(\overline{\lambda v}\big) \\ &= \overline{\lambda}v^{T}\overline{v} &\qquad (1) \\\\ (Mv)^{T} \overline{v} &= \lambda v^{T} \overline{v}. &\qquad (2) \end{aligned}$

On comparing equations (1) and (2), it's clear that $\ \overline{\lambda} = \lambda,$ so $\lambda \in \mathbb{R}.\ _\square$

Proof by Induction

For $n = 1,$ $M$ and $v$ are scalars, so $Mv = \lambda v,$ where $M = \lambda.$ Thus, we can pick any non-zero real scalar $v$ to form a basis for $\mathbb{R}.$

Induction Hypothesis:
Every $\ k \times k$ symmetric matrix is diagonalisable for $k = 1,...,n-1.$ So, let $M \in M_{n} (\mathbb{R})$ be symmetric. We break the induction part into 3 steps:

• Step 1:
  We can choose eigenvalue $\lambda_{1} \in \mathbb{R}$ by the above lemma (see comment for details), and let's normalize a corresponding eigenvector such that $\ |v_{1}| = 1$. Now we can extend to a basis $\ u_{1}, u_{2}, ... , u_{n}$ for $\mathbb{R^n}$ and apply the Gram-Schmidt procedure to obtain an orthonormal basis $\ B = {v_{1}, v_{2}, ... , v_{n}}$ for $\mathbb{R^n}$.

• Step 2:
  Define $P= [v_{1} v_{2} \ldots v_{n}]$. Now the columns of $P$ are orthonormal, so $P$ is an orthogonal matrix. That is, $P^{T} = P^{-1}$. Also, define $\ A= P^{-1}MP = P^{T}MP,$ then $A^{T} = \left(\big(P^{T}M\big)P\right)^{T} = P^{T}\big(P^{T}M\big)^{T} = P^{T}M^{T}P = P^{T}MP = A,$ so $A$ is symmetric. Now we use this to show how $A$ can be written in block matrix form. So, pre-multiplying the standard basis vector $\mathbf{e_{1}}$ by $A$ gives $\ P^{T}AP\mathbf{e_{1}} = P^{T}A v_{1} = \lambda_{1}P^{T}v_{1} = \lambda_{1} [v_{1}]_{B} = \lambda_{1} \begin{bmatrix} 1 \\ 0 \\ \vdots \\ 0 \end{bmatrix}_{B}.$ So, $A$ has the matrix block form $\ \begin{bmatrix} \lambda_{1} \quad 0 \\ 0 \quad C \end{bmatrix},$ where $C \in M_{n-1}(\mathbb{R})$ is symmetric.

• Step 3:
  Now, by the induction hypothesis, there exists a diagonal matrix $D$ and orthogonal matrix $Q$ such that $D= Q^{-1} CQ = Q^{T} CQ$. Now define $R = P \begin{bmatrix} 1 \quad 0 \\ 0 \quad Q \end{bmatrix}.$ We aim to show that $R$ is orthogonal and $\ R^{T} MR$ is diagonal. For the first part, $\begin{bmatrix} 1 \quad 0 \\ 0 \quad Q \end{bmatrix}^{-1} = \begin{bmatrix}1 \quad 0 \\ 0 \quad Q^{-1} \end{bmatrix} \implies R^{-1} = \begin{bmatrix}1 \quad 0 \\ 0 \quad Q^{-1} \end{bmatrix} P^{-1} = \begin{bmatrix}1 \quad 0 \\ 0 \quad Q^{T} \end{bmatrix} P^{T} = R^{T}.$ Finally, $R^{T}MR = \begin{bmatrix} 1 \quad 0 \\ 0 \quad Q^{T} \end{bmatrix} P^{T}MP \begin{bmatrix}1 \quad 0 \\ 0 \quad Q \end{bmatrix} = \begin{bmatrix} \lambda_{1} \quad 0 \\ 0 \quad D \end{bmatrix}.$ So there exists an orthogonal matrix $R$ such that $\ R^{T}MR$ is diagonal, which is equivalent to saying that there exists an orthonormal basis for $\mathbb{R^n}$ consisting of eigenvalues of $M.\ _\square$

(Note: The proof is yet to be peer-reviewed, so it may contain errors.)