Chapter 4 Exercises and Solutions

Twelve exercises, worked in full. Every eigenvalue, eigenvector and singular value below was checked against NumPy, and every eigenvector reported here satisfies $\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ to 0.0e+00 — exactly, not approximately, because they are all integer vectors.

The last two are proofs rather than computations. Both are stated as things to show, and both are used earlier in the chapter, so they are worth doing rather than skipping.

How to use this page

Try each exercise before reading on. The determinants (4.1, 4.2) are arithmetic drills. The eigenspace problems (4.3–4.7) are the bulk of the chapter and repay doing by hand — the point of each is a different way the count of eigenvectors can come up short. The SVDs (4.8, 4.9) follow §4.5.2’s three steps mechanically. 4.10 is one line once you have 4.8. And 4.11 and 4.12 are the two facts the chapter used without proving.

A shorthand used throughout

For each eigenvalue I give the algebraic multiplicity (its multiplicity as a root of the characteristic polynomial) and the geometric multiplicity (the dimension of its eigenspace, $\dim\ker(\mathbf{A}-\lambda\mathbf{I})$). The matrix is diagonalizable exactly when the geometric multiplicities sum to $n$ — Definition 4.13 and Theorem 4.20.

4.1 — Determinant by Laplace expansion and by Sarrus

Compute the determinant using the Laplace expansion (using the first row) and the Sarrus rule for

$$\mathbf{A} = \begin{bmatrix}1 & 3 & 5\\ 2 & 4 & 6\\ 0 & 2 & 4\end{bmatrix}$$

Laplace on the first row (Theorem 4.2):

$$\det\mathbf{A} = 1\begin{vmatrix}4&6\\2&4\end{vmatrix} - 3\begin{vmatrix}2&6\\0&4\end{vmatrix} + 5\begin{vmatrix}2&4\\0&2\end{vmatrix} = 1(16-12) - 3(8-0) + 5(4-0) = 4 - 24 + 20 = \boxed{0}$$

Sarrus (Equation 4.6) — the three down-right products minus the three down-left products:

$$\det\mathbf{A} = (1\cdot4\cdot4 + 3\cdot6\cdot0 + 5\cdot2\cdot2) - (0\cdot4\cdot5 + 2\cdot6\cdot1 + 4\cdot2\cdot3) = (16 + 0 + 20) - (0 + 12 + 24) = 36 - 36 = 0$$

Both give $0$, so the matrix is singular and has rank $2$.

Why it is zero. There is an exact linear dependence among the rows:

$$\mathbf{r}_3 = 2\mathbf{r}_1 - \mathbf{r}_2 \quad\text{since}\quad 2(1,3,5) - (2,4,6) = (0,2,4)$$

That is the interesting part of the answer. A determinant of zero is not a computation that failed; it is the statement that one row carries no information the other two did not already have.

4.2 — A 5 by 5 determinant, efficiently

$$\mathbf{A} = \begin{bmatrix} 2 & 0 & 1 & 2 & 0\\ 2 & -1 & 0 & 1 & 1\\ 0 & 1 & 2 & 1 & 2\\ -2 & 0 & 2 & -1 & 2\\ 2 & 0 & 0 & 1 & 1 \end{bmatrix}$$

“Efficiently” is the whole exercise. Laplace expansion on a $5\times5$ costs $5! = 120$ terms; Gaussian elimination costs about $5^3/3 \approx 42$ multiplications. Use row operations, which leave the determinant unchanged (Theorem 4.3), and read off the product of the pivots.

The second column is the place to start: it has only two nonzero entries, so expanding there — or eliminating into it — is cheapest. Working the elimination through gives

$$\det\mathbf{A} = \boxed{6}$$

Verified exactly, not just numerically: running Gaussian elimination in rational arithmetic with fractions.Fraction returns the integer $6$ with no rounding anywhere, and np.linalg.det returns $6.0000000000$.

The general lesson

Every property in §4.1’s list of seven exists to let you avoid the definition. Row operations preserve the determinant, a triangular matrix’s determinant is the product of its diagonal, and swapping two rows flips the sign. Those three facts turn any determinant into an elimination.

4.3 — Eigenspaces of two 2 by 2 matrices

(a) $\begin{bmatrix}1&0\\1&1\end{bmatrix}$

$$\det(\mathbf{A}-\lambda\mathbf{I}) = \begin{vmatrix}1-\lambda & 0\\ 1 & 1-\lambda\end{vmatrix} = (1-\lambda)^2$$

So $\lambda = 1$ with algebraic multiplicity 2. The eigenspace:

$$(\mathbf{A}-\mathbf{I})\mathbf{x} = \begin{bmatrix}0&0\\1&0\end{bmatrix}\begin{bmatrix}x_1\\x_2\end{bmatrix} = \mathbf{0} \implies x_1 = 0$$ $$E_1 = \mathrm{span}\left\{\begin{bmatrix}0\\1\end{bmatrix}\right\}, \qquad \text{geometric multiplicity } 1$$

One eigenvector for a $2\times2$ matrix, so this matrix is defective and not diagonalizable. Note $\det\mathbf{A} = 1 \neq 0$, so it is nevertheless invertible — this is 4.5’s point in miniature.

(b) $\begin{bmatrix}-2&2\\2&1\end{bmatrix}$

$$\det(\mathbf{A}-\lambda\mathbf{I}) = (-2-\lambda)(1-\lambda) - 4 = \lambda^2 + \lambda - 6 = (\lambda-2)(\lambda+3)$$

so $\lambda_1 = 2$, $\lambda_2 = -3$. Check against Equations 4.23–4.24: trace $= -1 = 2 + (-3)$ ✓, and $\det = -6 = 2\cdot(-3)$ ✓.

$$E_2 = \mathrm{span}\left\{\begin{bmatrix}1\\2\end{bmatrix}\right\}, \qquad E_{-3} = \mathrm{span}\left\{\begin{bmatrix}2\\-1\end{bmatrix}\right\}$$

Distinct eigenvalues, so by Theorem 4.12 the eigenvectors are independent: diagonalizable. And the matrix is symmetric, so the spectral theorem promises the eigenvectors are orthogonal — check $(1,2)\cdot(2,-1) = 0$ ✓.

4.4 — All eigenspaces of a 4 by 4

$$\mathbf{A} = \begin{bmatrix}0&-1&1&1\\ -1&1&-2&3\\ 2&-1&0&0\\ 1&-1&1&0\end{bmatrix}$$

$\det\mathbf{A} = 2$ and $\mathrm{tr}\,\mathbf{A} = 1$. The spectrum is

$\lambda$	algebraic	geometric	eigenspace
$2$	1	1	$\mathrm{span}\{(1,0,1,1)^\top\}$
$1$	1	1	$\mathrm{span}\{(1,1,1,1)^\top\}$
$-1$	2	1	$\mathrm{span}\{(0,1,1,0)^\top\}$

Sanity checks: the eigenvalues sum to $2 + 1 - 1 - 1 = 1 = \mathrm{tr}\,\mathbf{A}$ ✓ and multiply to $2\cdot1\cdot(-1)\cdot(-1) = 2 = \det\mathbf{A}$ ✓ (Theorems 4.16, 4.17).

Geometric multiplicities sum to $3 < 4$, so $\mathbf{A}$ is defective. The culprit is $\lambda = -1$, which appears twice as a root but contributes only a one-dimensional eigenspace.

All three eigenvectors are exact integer vectors — each satisfies $\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ to 0.0e+00.

4.5 — Diagonalizability is unrelated to invertibility

Determine for the following four matrices whether they are diagonalizable and/or invertible:

$$\begin{bmatrix}1&0\\0&1\end{bmatrix},\quad \begin{bmatrix}1&0\\0&0\end{bmatrix},\quad \begin{bmatrix}1&1\\0&1\end{bmatrix},\quad \begin{bmatrix}0&1\\0&0\end{bmatrix}$$

This is the exercise that makes §4.7’s warning concrete. All four combinations occur, one per matrix, in exactly this order:

matrix	$\det$	spectrum	$\sum$ geometric	invertible	diagonalizable
$\begin{bmatrix}1&0\\0&1\end{bmatrix}$	$1$	$\lambda=1$, algebraic $2$, geometric $2$	$2$	yes	yes
$\begin{bmatrix}1&0\\0&0\end{bmatrix}$	$0$	$\lambda = 1, 0$, each geometric $1$	$2$	no	yes
$\begin{bmatrix}1&1\\0&1\end{bmatrix}$	$1$	$\lambda=1$, algebraic $2$, geometric $1$	$1$	yes	no
$\begin{bmatrix}0&1\\0&0\end{bmatrix}$	$0$	$\lambda=0$, algebraic $2$, geometric $1$	$1$	no	no

Read the last two columns: yes yes, no yes, yes no, no no. A $2\times2$ example for each cell of a $2\times2$ table.

The mechanism differs in the two “no” rows for diagonalizability, though: both are Jordan blocks with a repeated eigenvalue and a deficient eigenspace. And note that the second matrix is diagonal, hence trivially diagonalizable, while being singular — a zero eigenvalue is still an eigenvalue, with a perfectly good eigenvector.

4.6 — Eigenspaces, and diagonalizability

(a) $\mathbf{A} = \begin{bmatrix}2&3&0\\1&4&3\\0&0&1\end{bmatrix}$

$\det\mathbf{A} = 5$, $\mathrm{tr}\,\mathbf{A} = 7$. The last row makes the characteristic polynomial factor immediately:

$$\det(\mathbf{A}-\lambda\mathbf{I}) = (1-\lambda)\bigl[(2-\lambda)(4-\lambda) - 3\bigr] = (1-\lambda)(\lambda^2 - 6\lambda + 5) = (1-\lambda)(\lambda-1)(\lambda-5)$$

so $\lambda = 5$ once and $\lambda = 1$ twice. Check: $5 + 1 + 1 = 7$ ✓ and $5\cdot1\cdot1 = 5$ ✓.

$$E_5 = \mathrm{span}\left\{\begin{bmatrix}1\\1\\0\end{bmatrix}\right\},\qquad E_1 = \mathrm{span}\left\{\begin{bmatrix}3\\-1\\0\end{bmatrix}\right\}$$

$1 + 1 = 2 < 3$: defective, not diagonalizable. Both eigenvectors have a zero third component, so the whole eigenspace picture lives in the $x_1x_2$ plane and nothing reaches the third dimension.

(b) $\mathbf{A} = \begin{bmatrix}1&1&0&0\\0&0&0&0\\0&0&0&0\\0&0&0&0\end{bmatrix}$

Rank $1$, so the kernel is three-dimensional. The characteristic polynomial is $-\lambda^3(1-\lambda)$: $\lambda = 1$ once and $\lambda = 0$ three times.

$$E_1 = \mathrm{span}\left\{(1,0,0,0)^\top\right\},\qquad E_0 = \mathrm{span}\left\{(1,-1,0,0)^\top,\ (0,0,1,0)^\top,\ (0,0,0,1)^\top\right\}$$

Geometric multiplicities $1 + 3 = 4$: diagonalizable, with $\mathbf{D} = \mathrm{diag}(1,0,0,0)$.

This is the pair to compare. (a) has a repeated eigenvalue and is defective; (b) has an eigenvalue repeated three times and is perfectly diagonalizable. A repeated eigenvalue is a warning, not a verdict — what matters is whether $\dim\ker(\mathbf{A}-\lambda\mathbf{I})$ keeps up.

4.7 — Diagonalizable? Give the diagonal form and a basis

(a) $\mathbf{A} = \begin{bmatrix}0&1\\-8&4\end{bmatrix}$

$$\det(\mathbf{A}-\lambda\mathbf{I}) = -\lambda(4-\lambda) + 8 = \lambda^2 - 4\lambda + 8$$

Discriminant $16 - 32 = -16 < 0$, so $\lambda = 2 \pm 2i$.

Not diagonalizable over $\mathbb{R}$ — there is no real eigenvector, because a real matrix with complex eigenvalues has no invariant line. Over $\mathbb{C}$ it is diagonalizable: two distinct eigenvalues, hence two independent eigenvectors, with $\mathbf{D} = \mathrm{diag}(2+2i,\ 2-2i)$.

This is the answer the book’s §4.7 warning is pointing at. $\det\mathbf{A} = 8 \neq 0$, so the matrix is invertible; invertibility and diagonalizability are simply different questions.

(b) $\mathbf{A} = \begin{bmatrix}1&1&1\\1&1&1\\1&1&1\end{bmatrix}$

Rank $1$: every row is the same. So $\lambda = 0$ has algebraic multiplicity $2$ and — since $\dim\ker\mathbf{A} = 3 - 1 = 2$ — geometric multiplicity $2$ as well. The remaining eigenvalue is the trace, $3$.

$$\mathbf{D} = \begin{bmatrix}3&0&0\\0&0&0\\0&0&0\end{bmatrix},\qquad \mathbf{P} = \begin{bmatrix}1 & 1 & 1\\ 1 & -1 & 0\\ 1 & 0 & -1\end{bmatrix}$$

Diagonalizable. The matrix is symmetric, so the spectral theorem guarantees it — and an orthonormal basis exists too: normalise $(1,1,1)$ and replace the two kernel vectors with an orthonormal pair inside that plane, for instance $(1,-1,0)/\sqrt2$ and $(1,1,-2)/\sqrt6$.

(c) $\mathbf{A} = \begin{bmatrix}5&4&2&1\\ 0&1&-1&-1\\ -1&-1&3&0\\ 1&1&-1&2\end{bmatrix}$

$\det\mathbf{A} = 32$, $\mathrm{tr}\,\mathbf{A} = 11$.

$\lambda$	algebraic	geometric	eigenvector
$4$	2	1	$(1,0,-1,1)^\top$
$2$	1	1	$(1,-1,0,1)^\top$
$1$	1	1	$(1,-1,0,0)^\top$

Check: $4 + 4 + 2 + 1 = 11$ ✓ and $4\cdot4\cdot2\cdot1 = 32$ ✓.

Geometric multiplicities sum to $3 < 4$: not diagonalizable, because $\lambda = 4$ is a double root with only a one-dimensional eigenspace.

(d) $\mathbf{A} = \begin{bmatrix}5&-6&-6\\ -1&4&2\\ 3&-6&-4\end{bmatrix}$

$\det\mathbf{A} = 4$, $\mathrm{tr}\,\mathbf{A} = 5$. Spectrum: $\lambda = 2$ twice and $\lambda = 1$ once. Check $2 + 2 + 1 = 5$ ✓ and $2\cdot2\cdot1 = 4$ ✓.

For $\lambda = 2$:

$$\mathbf{A} - 2\mathbf{I} = \begin{bmatrix}3&-6&-6\\ -1&2&2\\ 3&-6&-6\end{bmatrix}$$

Every row is a multiple of $(1,-2,-2)$, so the matrix has rank $1$ and the eigenspace is the plane $x_1 - 2x_2 - 2x_3 = 0$, which is two-dimensional:

$$E_2 = \mathrm{span}\left\{\begin{bmatrix}2\\1\\0\end{bmatrix},\ \begin{bmatrix}2\\0\\1\end{bmatrix}\right\}, \qquad E_1 = \mathrm{span}\left\{\begin{bmatrix}3\\-1\\3\end{bmatrix}\right\}$$

$2 + 1 = 3$: diagonalizable, with

$$\mathbf{D} = \begin{bmatrix}2&0&0\\0&2&0\\0&0&1\end{bmatrix},\qquad \mathbf{P} = \begin{bmatrix}2&2&3\\ 1&0&-1\\ 0&1&3\end{bmatrix}$$

Compare (c) and (d). Both have a double eigenvalue; in (c) the rank of $\mathbf{A}-\lambda\mathbf{I}$ is $3$, leaving nullity $1$; in (d) it is $1$, leaving nullity $2$. That single number is the difference between defective and diagonalizable.

The basis for a repeated eigenvalue is not unique

Any two independent vectors in the plane $x_1 - 2x_2 - 2x_3 = 0$ will do for $E_2$ in part (d), so different textbooks and different LAPACK builds report different $\mathbf{P}$. NumPy returns $(0.5727, 1, -0.7137)$ and $(1, -0.1260, 0.6260)$ for that eigenspace — both in the plane, neither matching the tidy integer pair. That is the reproducibility hazard from §4.4’s pitfalls, met in the wild.

4.8 — Find the SVD of a 2 by 3

$$\mathbf{A} = \begin{bmatrix}3&2&2\\ 2&3&-2\end{bmatrix}$$

Step 1 — the smaller Gram matrix. Here $\mathbf{A}\mathbf{A}^\top$ is $2\times2$ and $\mathbf{A}^\top\mathbf{A}$ is $3\times3$, so start with the small one:

$$\mathbf{A}\mathbf{A}^\top = \begin{bmatrix}17 & 8\\ 8 & 17\end{bmatrix}$$

Its eigenvalues are $17 \pm 8$, that is $25$ and $9$, with eigenvectors $(1,1)$ and $(1,-1)$ — a $2\times2$ matrix of the form $\begin{bmatrix}a&b\\b&a\end{bmatrix}$ always has those. So

$$\sigma_1 = \sqrt{25} = 5,\qquad \sigma_2 = \sqrt{9} = 3,\qquad \mathbf{u}_1 = \tfrac{1}{\sqrt2}\begin{bmatrix}1\\1\end{bmatrix},\qquad \mathbf{u}_2 = \tfrac{1}{\sqrt2}\begin{bmatrix}1\\-1\end{bmatrix}$$

Step 2 — the right-singular vectors from $\mathbf{v}_i = \mathbf{A}^\top\mathbf{u}_i/\sigma_i$:

$$\mathbf{v}_1 = \tfrac{1}{\sqrt2}\begin{bmatrix}1\\1\\0\end{bmatrix},\qquad \mathbf{v}_2 = \tfrac{1}{\sqrt{18}}\begin{bmatrix}1\\-1\\4\end{bmatrix},\qquad \mathbf{v}_3 = \tfrac{1}{3}\begin{bmatrix}2\\-2\\-1\end{bmatrix}$$

$\mathbf{v}_3$ comes from the kernel: $\sigma_3 = 0$ and $\mathbf{A}\mathbf{v}_3 = \mathbf{0}$.

The decomposition.

$$\mathbf{A} = \underbrace{\frac{1}{\sqrt2}\begin{bmatrix}1&1\\1&-1\end{bmatrix}}_{\mathbf{U}} \underbrace{\begin{bmatrix}5&0&0\\0&3&0\end{bmatrix}}_{\boldsymbol{\Sigma}} \underbrace{\begin{bmatrix} \tfrac{1}{\sqrt2} & \tfrac{1}{\sqrt2} & 0\\[2pt] \tfrac{1}{\sqrt{18}} & -\tfrac{1}{\sqrt{18}} & \tfrac{4}{\sqrt{18}}\\[2pt] \tfrac{2}{3} & -\tfrac{2}{3} & -\tfrac{1}{3} \end{bmatrix}}_{\mathbf{V}^\top}$$

Verified from these exact vectors: $\lVert\mathbf{A}\mathbf{v}_1\rVert = 5.000000$, $\lVert\mathbf{A}\mathbf{v}_2\rVert = 3.000000$, $\lVert\mathbf{A}\mathbf{v}_3\rVert = 1.11\times10^{-16}$, $\mathbf{V}$ orthogonal to $2.22\times10^{-16}$, and the reconstruction $\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top$ differing from $\mathbf{A}$ by $4.44\times10^{-16}$.

Start with the smaller Gram matrix

$\mathbf{A}^\top\mathbf{A}$ here is $\begin{bmatrix}13&12&2\\12&13&-2\\2&-2&8\end{bmatrix}$ — a $3\times3$ whose characteristic polynomial you would have to factor. $\mathbf{A}\mathbf{A}^\top$ is $2\times2$ with eigenvalues you can read off. They share their nonzero eigenvalues (that is 4.11), so always start with whichever is smaller and get the other factor from $\mathbf{v}_i = \mathbf{A}^\top\mathbf{u}_i/\sigma_i$.

4.9 — Find the SVD of a 2 by 2

$$\mathbf{A} = \begin{bmatrix}2&2\\ -1&1\end{bmatrix}$$ $$\mathbf{A}^\top\mathbf{A} = \begin{bmatrix}5&3\\3&5\end{bmatrix}$$

Again of the form $\begin{bmatrix}a&b\\b&a\end{bmatrix}$, so eigenvalues $5\pm3 = 8, 2$ with eigenvectors $(1,1)$ and $(-1,1)$. Hence

$$\sigma_1 = \sqrt8 = 2\sqrt2 \approx 2.828427,\qquad \sigma_2 = \sqrt2 \approx 1.414214$$ $$\mathbf{v}_1 = \tfrac{1}{\sqrt2}\begin{bmatrix}1\\1\end{bmatrix},\qquad \mathbf{v}_2 = \tfrac{1}{\sqrt2}\begin{bmatrix}-1\\1\end{bmatrix}$$

Now the left-singular vectors, and they come out remarkably clean:

$$\mathbf{u}_1 = \frac{\mathbf{A}\mathbf{v}_1}{\sigma_1} = \frac{1}{2\sqrt2}\begin{bmatrix}2\sqrt2\\0\end{bmatrix} = \begin{bmatrix}1\\0\end{bmatrix}, \qquad \mathbf{u}_2 = \frac{\mathbf{A}\mathbf{v}_2}{\sigma_2} = \frac{1}{\sqrt2}\begin{bmatrix}0\\\sqrt2\end{bmatrix} = \begin{bmatrix}0\\1\end{bmatrix}$$

So $\mathbf{U} = \mathbf{I}$ and

$$\mathbf{A} = \mathbf{I}\begin{bmatrix}2\sqrt2 & 0\\ 0 & \sqrt2\end{bmatrix} \frac{1}{\sqrt2}\begin{bmatrix}1&1\\ -1&1\end{bmatrix}$$

Reconstruction gap $4.44\times10^{-16}$. Geometrically: $\mathbf{A}$ rotates by $-45°$ and then stretches along the coordinate axes, so no second rotation is needed. Note this is the reduced-form convention of Equation 4.89 and the full form at once, since the matrix is square and full rank.

4.10 — The best rank-1 approximation

Find the best rank-$1$ approximation of $\mathbf{A} = \begin{bmatrix}3&2&2\\ 2&3&-2\end{bmatrix}$.

By Eckart-Young (Theorem 4.25) the answer is the first term of the SVD sum, and 4.8 already computed it:

$$\hat{\mathbf{A}}(1) = \sigma_1\mathbf{u}_1\mathbf{v}_1^\top = 5 \cdot \frac{1}{\sqrt2}\begin{bmatrix}1\\1\end{bmatrix} \cdot \frac{1}{\sqrt2}\begin{bmatrix}1&1&0\end{bmatrix} = \frac{5}{2}\begin{bmatrix}1&1&0\\1&1&0\end{bmatrix} = \begin{bmatrix}2.5&2.5&0\\ 2.5&2.5&0\end{bmatrix}$$

And the error is known in advance, by Equation 4.95:

$$\bigl\lVert\mathbf{A}-\hat{\mathbf{A}}(1)\bigr\rVert_2 = \sigma_2 = 3$$

Measured: $3.000000$ in both the spectral and the Frobenius norm. They agree here because the residual $\mathbf{A}-\hat{\mathbf{A}}(1)$ has only one nonzero singular value, and for a rank-1 matrix the two norms coincide.

The third column of $\hat{\mathbf{A}}(1)$ is zero, which is worth a moment: $\mathbf{v}_1$ has a zero third component, so the best rank-1 approximation of this matrix simply cannot represent the third feature at all. That is the geometry of “keep the top direction” — it is a choice about which columns survive as much as about which magnitudes do.

4.11 — Both Gram matrices share their nonzero eigenvalues

Show that for any $\mathbf{A} \in \mathbb{R}^{m\times n}$ the matrices $\mathbf{A}^\top\mathbf{A}$ and $\mathbf{A}\mathbf{A}^\top$ possess the same nonzero eigenvalues.

Proof. Let $\lambda \neq 0$ be an eigenvalue of $\mathbf{A}^\top\mathbf{A}$ with eigenvector $\mathbf{v} \neq \mathbf{0}$:

$$\mathbf{A}^\top\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$$

Multiply on the left by $\mathbf{A}$:

$$\mathbf{A}\mathbf{A}^\top(\mathbf{A}\mathbf{v}) = \lambda(\mathbf{A}\mathbf{v})$$

So $\mathbf{A}\mathbf{v}$ is an eigenvector of $\mathbf{A}\mathbf{A}^\top$ with the same eigenvalue $\lambda$ — provided $\mathbf{A}\mathbf{v} \neq \mathbf{0}$. And it is nonzero, because

$$\lVert\mathbf{A}\mathbf{v}\rVert^2 = \mathbf{v}^\top\mathbf{A}^\top\mathbf{A}\mathbf{v} = \lambda\lVert\mathbf{v}\rVert^2 \neq 0$$

using $\lambda \neq 0$ and $\mathbf{v} \neq \mathbf{0}$. So every nonzero eigenvalue of $\mathbf{A}^\top\mathbf{A}$ is one of $\mathbf{A}\mathbf{A}^\top$. Swapping the roles of $\mathbf{A}$ and $\mathbf{A}^\top$ gives the converse, and the two sets are equal. $\blacksquare$

Where $\lambda \neq 0$ is essential. The zero eigenvalues need not match in multiplicity: for a $5\times3$ matrix of rank $3$, $\mathbf{A}^\top\mathbf{A}$ is $3\times3$ with no zero eigenvalues at all while $\mathbf{A}\mathbf{A}^\top$ is $5\times5$ with two. The counts differ by exactly $\lvert m-n\rvert$ plus the rank deficiency, and the map $\mathbf{v} \mapsto \mathbf{A}\mathbf{v}$ collapses precisely on the kernel.

Measured. Over $4000$ random rectangular matrices with shapes drawn from $2\times2$ up to $7\times7$, the worst disagreement between the top $\min(m,n)$ eigenvalues of the two Gram matrices was $4.97\times10^{-14}$. Over $2000$ deliberately rank-deficient matrices the worst relative disagreement was $2.11\times10^{-15}$.

This is the fact §4.5.2 used without proof to argue that both Gram matrices produce the same $\boldsymbol{\Sigma}$.

4.12 — The spectral norm is the largest singular value

Show that for $\mathbf{x} \neq \mathbf{0}$, Theorem 4.24 holds:

$$\max_{\mathbf{x}} \frac{\lVert\mathbf{A}\mathbf{x}\rVert_2}{\lVert\mathbf{x}\rVert_2} = \sigma_1$$

Proof. The ratio is scale-invariant, so restrict to $\lVert\mathbf{x}\rVert_2 = 1$. Substitute the SVD:

$$\lVert\mathbf{A}\mathbf{x}\rVert_2 = \lVert\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top\mathbf{x}\rVert_2 = \lVert\boldsymbol{\Sigma}\mathbf{V}^\top\mathbf{x}\rVert_2$$

because $\mathbf{U}$ is orthogonal and orthogonal matrices preserve the Euclidean norm (§3.4). Write $\mathbf{y} = \mathbf{V}^\top\mathbf{x}$; since $\mathbf{V}^\top$ is also orthogonal, $\lVert\mathbf{y}\rVert_2 = \lVert\mathbf{x}\rVert_2 = 1$, and as $\mathbf{x}$ ranges over the unit sphere so does $\mathbf{y}$. So the problem becomes

$$\max_{\lVert\mathbf{y}\rVert_2 = 1} \lVert\boldsymbol{\Sigma}\mathbf{y}\rVert_2 = \max_{\lVert\mathbf{y}\rVert_2 = 1} \sqrt{\sum_{i} \sigma_i^2 y_i^2}$$

Since $\sigma_1 \geq \sigma_i$ for every $i$ and $\sum_i y_i^2 = 1$,

$$\sum_i \sigma_i^2 y_i^2 \leq \sigma_1^2\sum_i y_i^2 = \sigma_1^2$$

so the ratio never exceeds $\sigma_1$. And the bound is attained: take $\mathbf{y} = \mathbf{e}_1$, that is $\mathbf{x} = \mathbf{v}_1$, giving $\lVert\mathbf{A}\mathbf{v}_1\rVert_2 = \sigma_1$. A maximum that is both an upper bound and achieved is the maximum. $\blacksquare$

The proof is really two observations: orthogonal matrices do not change lengths, so the outer factors are irrelevant to the question; and a diagonal matrix stretches a unit vector most when the vector points along its largest entry.

Measured. $1{,}200{,}000$ random unit vectors across $300$ random matrices: the ratio $\lVert\mathbf{A}\mathbf{x}\rVert/\lVert\mathbf{x}\rVert$ exceeded $\sigma_1$ exactly $0$ times. And for exercise 4.8’s matrix, $\lVert\mathbf{A}\mathbf{v}_1\rVert = 5.000000 = \sigma_1$ exactly.

Random sampling finds the bound but not the maximiser

In that experiment the largest shortfall of the sampled maximum below $\sigma_1$ was $0.164915$ — random unit vectors in five or six dimensions simply do not land near $\mathbf{v}_1$. So the sampling confirms the inequality convincingly and confirms attainment not at all. Attainment needs the argument, or $\mathbf{v}_1$ itself. This is a good habit to keep: a Monte-Carlo check can only ever verify one direction of a max.

The whole chapter, in one table

#	question	answer
4.1	$\det$ by Laplace and Sarrus	$0$; singular, rank $2$, because $\mathbf{r}_3 = 2\mathbf{r}_1 - \mathbf{r}_2$
4.2	a $5\times5$ determinant	$6$, exactly, by elimination rather than expansion
4.3	two sets of eigenspaces	(a) $\lambda=1$, algebraic $2$, geometric $1$ → defective; (b) $\lambda = 2, -3$ → diagonalizable and orthogonal
4.4	all eigenspaces of a $4\times4$	$\lambda = 2, 1, -1$ with $-1$ doubled but geometric $1$ → defective
4.5	diagonalizable vs invertible	all four combinations occur, one per matrix
4.6	eigenspaces; diagonalizable?	(a) $\lambda = 5, 1, 1$ geometric $1+1 = 2 < 3$ → defective; (b) $\lambda = 1, 0, 0, 0$ geometric $1+3 = 4$ → diagonalizable
4.7	four diagonalizability questions	(a) complex $2\pm2i$: no over $\mathbb{R}$; (b) yes, $\mathbf{D} = \mathrm{diag}(3,0,0)$; (c) $\lambda=4$ doubled, geometric $1$ → no; (d) yes, $\mathbf{D} = \mathrm{diag}(2,2,1)$
4.8	SVD of a $2\times3$	$\sigma = 5, 3$; $\mathbf{U} = \tfrac{1}{\sqrt2}\begin{bmatrix}1&1\\1&-1\end{bmatrix}$
4.9	SVD of a $2\times2$	$\sigma = 2\sqrt2, \sqrt2$; $\mathbf{U} = \mathbf{I}$
4.10	best rank-1	$\tfrac52\begin{bmatrix}1&1&0\\1&1&0\end{bmatrix}$, error $3 = \sigma_2$
4.11	the two Gram matrices	same nonzero eigenvalues; the zero ones need not match in count
4.12	$\max\lVert\mathbf{A}\mathbf{x}\rVert/\lVert\mathbf{x}\rVert$	$\sigma_1$, attained at $\mathbf{x} = \mathbf{v}_1$

Six of the twelve are about one thing: how many independent eigenvectors are there? 4.3a, 4.4, 4.6a and 4.7c come up short; 4.6b and 4.7d do not, despite repeated eigenvalues. That is the chapter’s central distinction, drilled six times.

pch.quizTag Check your understanding

1. Exercises 4.6a and 4.6b both have a repeated eigenvalue. Why is one defective and the other not?

   A 4.6b is larger B Because defectiveness is about the geometric multiplicity, not the repeat: in 4.6a the double eigenvalue 1 has a one-dimensional eigenspace, while in 4.6b the triple eigenvalue 0 has a three-dimensional one, so the totals are 2 of 3 and 4 of 4 C Because 4.6b is singular D Because 4.6a is not symmetric
   pch.quizShowAnswer

   B — Because defectiveness is about the geometric multiplicity, not the repeat: in 4.6a the double eigenvalue 1 has a one-dimensional eigenspace, while in 4.6b the triple eigenvalue 0 has a three-dimensional one, so the totals are 2 of 3 and 4 of 4 — A repeated eigenvalue is a warning, not a verdict. 4.7d makes the same point: a doubled eigenvalue 2 whose A minus 2I has rank 1, leaving a two-dimensional eigenspace, so it is diagonalizable.

2. In 4.8, why start from A A-transpose rather than A-transpose A?

   A They give different answers B Because A is 2x3, so A A-transpose is only 2x2 with eigenvalues you can read off, whereas A-transpose A is 3x3 and needs a cubic factored — and by 4.11 they share their nonzero eigenvalues anyway C Because A-transpose A is not symmetric D Because only A A-transpose is positive definite
   pch.quizShowAnswer

   B — Because A is 2x3, so A A-transpose is only 2x2 with eigenvalues you can read off, whereas A-transpose A is 3x3 and needs a cubic factored — and by 4.11 they share their nonzero eigenvalues anyway — Then get the other factor from v-i equals A-transpose u-i over sigma-i. Exercise 4.11 is exactly the licence for this shortcut, which is why the book puts it in the same exercise set.

3. The proof of 4.11 needs lambda nonzero at one specific step. Which, and what breaks without it?

   A The first multiplication by A B Showing A v is not the zero vector: its squared norm is lambda times the squared norm of v, so it is nonzero only when lambda is. With lambda zero, A v can be zero and there is no eigenvector to carry across C The transpose identity D Nothing; the restriction is cosmetic
   pch.quizShowAnswer

   B — Showing A v is not the zero vector: its squared norm is lambda times the squared norm of v, so it is nonzero only when lambda is. With lambda zero, A v can be zero and there is no eigenvector to carry across — And this is not a technicality: for a 5x3 matrix of rank 3, A-transpose A has no zero eigenvalues while A A-transpose has two. The nonzero spectra match; the zero ones do not.

4. A Monte-Carlo run of 1.2 million unit vectors never exceeded sigma_1, but the sampled maximum fell short by up to 0.165. What does that experiment establish?

   A That the theorem is false B Only the inequality. It gives strong evidence that the ratio never exceeds sigma_1, and essentially no evidence that sigma_1 is attained — random vectors in five or six dimensions do not land near v_1. Attainment needs the argument, or v_1 itself C Both halves of the theorem D That more samples are needed
   pch.quizShowAnswer

   B — Only the inequality. It gives strong evidence that the ratio never exceeds sigma_1, and essentially no evidence that sigma_1 is attained — random vectors in five or six dimensions do not land near v_1. Attainment needs the argument, or v_1 itself — A general habit: sampling can refute an upper bound and can support one, but it cannot demonstrate that a maximum is achieved. Evaluating at the claimed maximiser can — ||A v_1|| came out to 5.000000 exactly for 4.8's matrix.

5. Exercise 4.10's answer has a zero third column. Is that a coincidence?

   A Yes, an artefact of the arithmetic B No. The best rank-1 approximation is sigma_1 times u_1 v_1-transpose, and v_1 is (1,1,0)/sqrt(2) — its zero third component forces a zero third column, so this approximation cannot represent the third feature at all C Yes, it depends on the sign convention D No, because the matrix is rank 2
   pch.quizShowAnswer

   B — No. The best rank-1 approximation is sigma_1 times u_1 v_1-transpose, and v_1 is (1,1,0)/sqrt(2) — its zero third component forces a zero third column, so this approximation cannot represent the third feature at all — Which is worth remembering when truncation is used for dimensionality reduction: keeping the top direction is a choice about which features survive, not only about which magnitudes do.

🧪 Try It Yourself

Exercise 1 – 4.1 and 4.2, two determinants

pch.exerciseTag pch.exerciseTitle

Exercise 2 – 4.3 to 4.7, every eigenspace

pch.exerciseTag pch.exerciseTitle

Exercise 3 – 4.4, 4.6, 4.7: the eigenvectors, exactly

pch.exerciseTag pch.exerciseTitle

Exercise 4 – 4.8, 4.9 and 4.10, from the exact vectors

pch.exerciseTag pch.exerciseTitle

Exercise 5 – 4.11 and 4.12, tested

pch.exerciseTag pch.exerciseTitle

Recall card

• 4.1: the determinant is $0$ because $\mathbf{r}_3 = 2\mathbf{r}_1 - \mathbf{r}_2$; Laplace and Sarrus agree, and rank is 2.
• 4.2: the $5\times5$ determinant is exactly $6$ — computed by elimination, about 42 multiplications against Laplace’s 120 terms.
• 4.3: (a) $\lambda=1$ with algebraic 2 and geometric 1, so defective despite being invertible; (b) $\lambda = 2, -3$ with orthogonal eigenvectors, since the matrix is symmetric.
• 4.4: $\lambda = 2, 1, -1$ with $-1$ doubled but geometric 1 — defective, 3 of 4. Trace and determinant confirm the spectrum.
• 4.5: all four invertible-by-diagonalizable combinations occur, one per matrix, in the order given.
• 4.6: (a) is defective with a doubled $\lambda=1$; (b) has $\lambda=0$ tripled and is still diagonalizable. A repeated eigenvalue is a warning, not a verdict.
• 4.7: (a) complex $2\pm2i$, so not diagonalizable over the reals though invertible; (b) and (d) diagonalizable; (c) defective because $\lambda=4$ is doubled with a one-dimensional eigenspace.
• 4.8: $\sigma = 5, 3$ — start from the $2\times2$ $\mathbf{A}\mathbf{A}^\top$, whose eigenvalues $17\pm8$ you can read off, then get $\mathbf{V}$ from $\mathbf{A}^\top\mathbf{u}_i/\sigma_i$.
• 4.9: $\sigma = 2\sqrt2, \sqrt2$ and $\mathbf{U} = \mathbf{I}$ — the matrix rotates then stretches along the coordinate axes.
• 4.10: the best rank-1 is $\tfrac52\begin{bmatrix}1&1&0\\1&1&0\end{bmatrix}$ with error exactly $\sigma_2 = 3$; its third column is zero because $\mathbf{v}_1$‘s third component is.
• 4.11: multiply $\mathbf{A}^\top\mathbf{A}\mathbf{v} = \lambda\mathbf{v}$ by $\mathbf{A}$; the step that needs $\lambda \neq 0$ is showing $\mathbf{A}\mathbf{v} \neq \mathbf{0}$. The zero eigenvalues differ in count — 0 against 2 for a $5\times3$ of rank 3.
• 4.12: orthogonal factors preserve length, so the problem reduces to maximising a diagonal stretch; the bound $\sigma_1$ is attained at $\mathbf{x} = \mathbf{v}_1$. Sampling verifies the inequality but never the attainment.

Next: Chapter 4 Formula Sheet — every definition, theorem and equation from the chapter on one page.