Chapter 3 Exercises and Solutions

Ten exercises, all of them worked. Try each before reading the solution — three of them (3.2, 3.5 and 3.6) are traps for a habit rather than tests of a formula, and they only teach you something if you fall into them first.

The verification code at the end of each solution is the one you should run. Every number on this page came out of it.

3.1 — Show that a formula is an inner product

Show that $\langle\cdot,\cdot\rangle$ defined for all $\mathbf{x} = [x_1, x_2]^\top \in \mathbb{R}^2$ and $\mathbf{y} = [y_1, y_2]^\top \in \mathbb{R}^2$ by

$\langle\mathbf{x},\mathbf{y}\rangle := x_1y_1 - (x_1y_2 + x_2y_1) + 2(x_2y_2)$

is an inner product.

Solution. Read off the matrix. Every term is a product $x_iy_j$ with a coefficient, and those coefficients are the entries of $\mathbf{A}$ in $\langle\mathbf{x},\mathbf{y}\rangle = \mathbf{x}^\top\mathbf{A}\mathbf{y}$:

term	coefficient	entry
$x_1y_1$	$1$	$A_{11} = 1$
$x_1y_2$	$-1$	$A_{12} = -1$
$x_2y_1$	$-1$	$A_{21} = -1$
$x_2y_2$	$2$	$A_{22} = 2$

$$\mathbf{A} = \begin{bmatrix}1 & -1\\ -1 & 2\end{bmatrix}$$

Now check the three conditions of Definition 3.3.

Bilinear. Automatic. Any expression of the form $\mathbf{x}^\top\mathbf{A}\mathbf{y}$ is linear in each argument separately, because matrix multiplication is.

Symmetric. $A_{12} = A_{21} = -1$, so $\mathbf{A} = \mathbf{A}^\top$ and $\langle\mathbf{x},\mathbf{y}\rangle = \langle\mathbf{y},\mathbf{x}\rangle$.

Positive definite. Complete the square:

$$\mathbf{x}^\top\mathbf{A}\mathbf{x} = x_1^2 - 2x_1x_2 + 2x_2^2 = (x_1 - x_2)^2 + x_2^2$$

A sum of two squares, so it is $\geq 0$ always. It equals zero only when both squares vanish: $x_2 = 0$ and $x_1 - x_2 = 0$, hence $x_1 = x_2 = 0$. So it is zero only at the origin. ∎

Completing the square is the argument to give, because it needs no eigenvalue machinery — Chapter 4 has not happened yet. If you want the eigenvalue check anyway, they are $0.381966$ and $2.618034$, both positive.

ex_3_1.py

import numpy as np
 
A = np.array([[1.0, -1.0], [-1.0, 2.0]])
print("symmetric:", np.allclose(A, A.T))
print("eigenvalues:", np.round(np.linalg.eigvalsh(A), 6))
 
x = np.array([3.0, -2.0])
print("form on (3, -2):", float(x @ A @ x))
print("(x1 - x2)^2 + x2^2 =", (3 - (-2)) ** 2 + (-2) ** 2)

output

symmetric: True
eigenvalues: [0.381966 2.618034]
form on (3, -2): 29.0
(x1 - x2)^2 + x2^2 = 29

3.2 — Is this one an inner product?

Consider $\mathbb{R}^2$ with $\langle\cdot,\cdot\rangle$ defined for all $\mathbf{x}$ and $\mathbf{y}$ in $\mathbb{R}^2$ as

$\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\top\underbrace{\begin{bmatrix}2 & 0\\ 1 & 2\end{bmatrix}}_{=:\mathbf{A}}\mathbf{y}$

Is $\langle\cdot,\cdot\rangle$ an inner product?

Solution. No. $\mathbf{A}$ is not symmetric: $A_{12} = 0$ while $A_{21} = 1$.

The one-line witness. Take $\mathbf{x} = \mathbf{e}_1$ and $\mathbf{y} = \mathbf{e}_2$:

$$\langle\mathbf{e}_1, \mathbf{e}_2\rangle = \mathbf{e}_1^\top\mathbf{A}\mathbf{e}_2 = A_{12} = 0, \qquad \langle\mathbf{e}_2, \mathbf{e}_1\rangle = A_{21} = 1$$

$0 \neq 1$, so symmetry fails and Definition 3.3 is not satisfied. ∎

Why this exercise is here. The quadratic form is perfectly well behaved:

$$\mathbf{x}^\top\mathbf{A}\mathbf{x} = 2x_1^2 + x_1x_2 + 2x_2^2$$

and only the symmetric part of $\mathbf{A}$ affects it, $\tfrac{1}{2}(\mathbf{A}+\mathbf{A}^\top) = \begin{bmatrix}2 & 0.5\\ 0.5 & 2\end{bmatrix}$, whose eigenvalues are $1.5$ and $2.5$ — both positive. So $\langle\mathbf{x},\mathbf{x}\rangle > 0$ for every nonzero $\mathbf{x}$, and a reader who tests only positive definiteness will pass this matrix.

The failure is entirely in the asymmetry, and its consequence is concrete: “the angle between $\mathbf{x}$ and $\mathbf{y}$” would depend on which one you named first.

ex_3_2.py

import numpy as np
 
A = np.array([[2.0, 0.0], [1.0, 2.0]])
e1, e2 = np.array([1.0, 0.0]), np.array([0.0, 1.0])
 
print("A   =", A.tolist())
print("A^T =", A.T.tolist())
print("symmetric:", bool(np.allclose(A, A.T)))
print("<e1, e2> =", float(e1 @ A @ e2), "  <e2, e1> =", float(e2 @ A @ e1))
 
sym = (A + A.T) / 2
print("symmetric part:", sym.tolist(), " eigenvalues:", np.round(np.linalg.eigvalsh(sym), 6))

output

A   = [[2.0, 0.0], [1.0, 2.0]]
A^T = [[2.0, 1.0], [0.0, 2.0]]
symmetric: False
<e1, e2> = 0.0   <e2, e1> = 1.0
symmetric part: [[2.0, 0.5], [0.5, 2.0]]  eigenvalues: [1.5 2.5]

The last line is the one to notice. The symmetric part has eigenvalues $1.5$ and $2.5$, both positive, so $\langle\mathbf{x},\mathbf{x}\rangle > 0$ for every nonzero $\mathbf{x}$. A verification routine that checks only positive definiteness — for instance by running Cholesky on $\tfrac12(\mathbf{A}+\mathbf{A}^\top)$, which is what several libraries silently do — reports this matrix as fine.

3.3 — Distance under two inner products

Compute the distance between $\mathbf{x} = [1, 2, 3]^\top$ and $\mathbf{y} = [-1, -1, 0]^\top$ using (a) $\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\top\mathbf{y}$ and (b) $\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\top\mathbf{A}\mathbf{y}$ with $\mathbf{A} := \begin{bmatrix}2&1&0\\1&3&-1\\0&-1&2\end{bmatrix}$.

Solution. Distance is the norm of the difference (Definition 3.6), so compute the difference once:

$$\mathbf{x} - \mathbf{y} = \begin{bmatrix}1-(-1)\\ 2-(-1)\\ 3-0\end{bmatrix} = \begin{bmatrix}2\\3\\3\end{bmatrix}$$

(a) $\lVert(2,3,3)\rVert^2 = 4 + 9 + 9 = 22$, so

$$d(\mathbf{x},\mathbf{y}) = \sqrt{22} \approx 4.690416 .$$

(b) First $\mathbf{A}(\mathbf{x}-\mathbf{y})$, row by row:

row	working	result
1	$2(2) + 1(3) + 0(3)$	$7$
2	$1(2) + 3(3) + (-1)(3)$	$8$
3	$0(2) + (-1)(3) + 2(3)$	$3$

Then $(2,3,3)\cdot(7,8,3) = 14 + 24 + 9 = 47$, so

$$d(\mathbf{x},\mathbf{y}) = \sqrt{47} \approx 6.855655 .$$

$\mathbf{A}$ really is an inner product: its eigenvalues are exactly $1$, $2$ and $4$. So both answers are correct and they differ by $46\%$ — which is the whole point of asking for both. ∎

ex_3_3.py

import numpy as np
 
x = np.array([1.0, 2.0, 3.0])
y = np.array([-1.0, -1.0, 0.0])
d = x - y
A = np.array([[2.0, 1.0, 0.0], [1.0, 3.0, -1.0], [0.0, -1.0, 2.0]])
 
print("x - y   =", d)
print("A(x-y)  =", A @ d)
print("(a) squared", float(d @ d), " distance", np.sqrt(float(d @ d)))
print("(b) squared", float(d @ A @ d), " distance", np.sqrt(float(d @ A @ d)))
print("A eigenvalues:", np.round(np.linalg.eigvalsh(A), 6))

output

x - y   = [2. 3. 3.]
A(x-y)  = [7. 8. 3.]
(a) squared 22.0  distance 4.69041575982343
(b) squared 47.0  distance 6.855654600401044
A eigenvalues: [1. 2. 4.]

3.4 — Angle under two inner products

Compute the angle between $\mathbf{x} = [1, 2]^\top$ and $\mathbf{y} = [-1, -1]^\top$ using (a) $\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\top\mathbf{y}$ and (b) $\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\top\mathbf{B}\mathbf{y}$ with $\mathbf{B} := \begin{bmatrix}2&1\\1&3\end{bmatrix}$.

Solution. Both parts use Equation 3.25, so the only work is three inner products each.

(a) Dot product.

$$\langle\mathbf{x},\mathbf{y}\rangle = 1(-1) + 2(-1) = -3, \qquad \lVert\mathbf{x}\rVert = \sqrt{1+4} = \sqrt5, \qquad \lVert\mathbf{y}\rVert = \sqrt{1+1} = \sqrt2$$ $$\cos\omega = \frac{-3}{\sqrt5\sqrt2} = \frac{-3}{\sqrt{10}} \approx -0.948683 \qquad\Longrightarrow\qquad \omega \approx 2.819842\ \text{rad} = 161.565051°$$

(b) With $\mathbf{B}$. Compute the matrix-vector products first, and reuse them:

$$\mathbf{B}\mathbf{x} = \begin{bmatrix}2(1)+1(2)\\ 1(1)+3(2)\end{bmatrix} = \begin{bmatrix}4\\7\end{bmatrix}, \qquad \mathbf{B}\mathbf{y} = \begin{bmatrix}2(-1)+1(-1)\\ 1(-1)+3(-1)\end{bmatrix} = \begin{bmatrix}-3\\-4\end{bmatrix}$$

quantity	working	value
$\langle\mathbf{x},\mathbf{y}\rangle$	$\mathbf{x}\cdot\mathbf{B}\mathbf{y} = 1(-3) + 2(-4)$	$-11$
$\langle\mathbf{x},\mathbf{x}\rangle$	$\mathbf{x}\cdot\mathbf{B}\mathbf{x} = 1(4) + 2(7)$	$18$
$\langle\mathbf{y},\mathbf{y}\rangle$	$\mathbf{y}\cdot\mathbf{B}\mathbf{y} = (-1)(-3) + (-1)(-4)$	$7$

$$\cos\omega = \frac{-11}{\sqrt{18}\sqrt{7}} = \frac{-11}{\sqrt{126}} \approx -0.979958 \qquad\Longrightarrow\qquad \omega \approx 2.941046\ \text{rad} = 168.509540°$$

$\mathbf{B}$‘s eigenvalues are $1.381966$ and $3.618034$, so it is a valid inner product. The two angles differ by nearly $7°$, and both are obtuse — these vectors point broadly opposite ways under either geometry, but by different amounts. ∎

ex_3_4.py

import numpy as np
 
x = np.array([1.0, 2.0])
y = np.array([-1.0, -1.0])
B = np.array([[2.0, 1.0], [1.0, 3.0]])
 
for name, M in (("dot", np.eye(2)), ("B  ", B)):
    ip = float(x @ M @ y)
    nx, ny = np.sqrt(float(x @ M @ x)), np.sqrt(float(y @ M @ y))
    c = ip / (nx * ny)
    print(f"({name}) <x,y> {ip:+.4f}  ||x|| {nx:.6f}  ||y|| {ny:.6f}  "
          f"cos {c:.6f}  omega {np.arccos(c):.6f} rad = {np.degrees(np.arccos(c)):.6f} deg")
 
print("-3/sqrt(10)  =", round(-3 / np.sqrt(10), 6))
print("-11/sqrt(126) =", round(-11 / np.sqrt(126), 6))
print("Bx =", B @ x, " By =", B @ y, " B eigenvalues:", np.round(np.linalg.eigvalsh(B), 6))

output

(dot) <x,y> -3.0000  ||x|| 2.236068  ||y|| 1.414214  cos -0.948683  omega 2.819842 rad = 161.565051 deg
(B  ) <x,y> -11.0000  ||x|| 4.242641  ||y|| 2.645751  cos -0.979958  omega 2.941046 rad = 168.509540 deg
-3/sqrt(10)  = -0.948683
-11/sqrt(126) = -0.979958
Bx = [4. 7.]  By = [-3. -4.]  B eigenvalues: [1.381966 3.618034]

3.5 — Projection in five dimensions

Consider the Euclidean vector space $\mathbb{R}^5$ with the dot product. A subspace $U \subseteq \mathbb{R}^5$ and $\mathbf{x} \in \mathbb{R}^5$ are given by

\qquad \mathbf{x} = \begin{bmatrix}-1\\-9\\-1\\4\\1\end{bmatrix}$$ **(a)** Determine the orthogonal projection $\pi_U(\mathbf{x})$ of $\mathbf{x}$ onto $U$. **(b)** Determine the distance $d(\mathbf{x}, U)$.

Solution. Read the margin note in §3.8.2 first. The problem gives a spanning set, and the book warns: if $U$ is given by spanning vectors that are not a basis, determine a basis before proceeding. Here that warning has teeth.

Step 0 — the rank. Four vectors, and

$$\mathrm{rank} = 3 .$$

One of them is redundant. Solving for it: the fourth vector is $1\cdot\mathbf{u}_1 + 2\cdot\mathbf{u}_2 + 1\cdot\mathbf{u}_3$. Check the first coordinate: $0 + 2(1) + (-3) = -1$ ✓; the second: $-1 + 2(-3) + 4 = -3$ ✓; the third: $2 + 2(1) + 1 = 5$ ✓; the fourth: $0 + 2(-1) + 2 = 0$ ✓; the fifth: $2 + 2(2) + 1 = 7$ ✓.

So $\dim U = 3$, and $\mathbf{B} = [\mathbf{u}_1\ \mathbf{u}_2\ \mathbf{u}_3]$ is a basis. Using all four columns makes $\mathbf{B}^\top\mathbf{B}$ singular and Equation 3.59 undefined — its condition number comes out at $2.0\times10^{17}$.

Step 1 — the normal equation on the three-column basis gives

$$\boldsymbol{\lambda} = \begin{bmatrix}-3\\4\\1\end{bmatrix} .$$

Step 2 — the projection.

$$\pi_U(\mathbf{x}) = \mathbf{B}\boldsymbol{\lambda} = -3\mathbf{u}_1 + 4\mathbf{u}_2 + \mathbf{u}_3 = \begin{bmatrix}1\\-5\\-1\\-2\\3\end{bmatrix}$$

Spot-check the first coordinate: $-3(0) + 4(1) + (-3) = 1$ ✓. The fifth: $-3(2) + 4(2) + 1 = -6 + 8 + 1 = 3$ ✓.

Step 3 — the distance.

$$\mathbf{x} - \pi_U(\mathbf{x}) = \begin{bmatrix}-2\\-4\\0\\6\\-2\end{bmatrix}, \qquad d(\mathbf{x}, U) = \sqrt{4 + 16 + 0 + 36 + 4} = \sqrt{60} \approx 7.745967$$

And the check: $\mathbf{U}^\top(\mathbf{x} - \pi_U(\mathbf{x})) = \mathbf{0}$ against all four spanning vectors, not merely the three in the basis — which it must be, since the fourth lies in the span of the others. ∎

ex_3_5.py

import numpy as np
 
U = np.array([[0.0, 1.0, -3.0, -1.0],
              [-1.0, -3.0, 4.0, -3.0],
              [2.0, 1.0, 1.0, 5.0],
              [0.0, -1.0, 2.0, 0.0],
              [2.0, 2.0, 1.0, 7.0]])
x = np.array([-1.0, -9.0, -1.0, 4.0, 1.0])
 
print("rank of the spanning set:", np.linalg.matrix_rank(U))
print("cond(U^T U):", f"{np.linalg.cond(U.T @ U):.3e}", "-> Eq 3.59 is undefined on all four columns")
print("column 4 in terms of the first three:",
      np.round(np.linalg.lstsq(U[:, :3], U[:, 3], rcond=None)[0], 6))
 
B = U[:, :3]                                  # a genuine basis
lam = np.linalg.solve(B.T @ B, B.T @ x)
proj = B @ lam
print("lambda      =", np.round(lam, 10))
print("(a) pi_U(x) =", np.round(proj, 10))
print("residual    =", np.round(x - proj, 10))
print("(b) distance =", round(float(np.linalg.norm(x - proj)), 6), " sqrt(60) =", round(float(np.sqrt(60)), 6))
print("orthogonal to all FOUR spanning vectors:", np.round(U.T @ (x - proj), 10))

output

rank of the spanning set: 3
cond(U^T U): 2.018e+17 -> Eq 3.59 is undefined on all four columns
column 4 in terms of the first three: [1. 2. 1.]
lambda      = [-3.  4.  1.]
(a) pi_U(x) = [ 1. -5. -1. -2.  3.]
residual    = [-2. -4.  0.  6. -2.]
(b) distance = 7.745967  sqrt(60) = 7.745967
orthogonal to all FOUR spanning vectors: [ 0.  0. -0.  0.]

3.6 — Projection under a non-standard inner product

Consider $\mathbb{R}^3$ with the inner product $\langle\mathbf{x},\mathbf{y}\rangle := \mathbf{x}^\top\begin{bmatrix}2&1&0\\1&2&-1\\0&-1&2\end{bmatrix}\mathbf{y}$. Furthermore, $\mathbf{e}_1, \mathbf{e}_2, \mathbf{e}_3$ are the standard basis in $\mathbb{R}^3$.

(a) Determine the orthogonal projection $\pi_U(\mathbf{e}_2)$ of $\mathbf{e}_2$ onto $U = \mathrm{span}[\mathbf{e}_1, \mathbf{e}_3]$. Hint: orthogonality is defined through the inner product. (b) Compute the distance $d(\mathbf{e}_2, U)$. (c) Draw the scenario: standard basis vectors and $\pi_U(\mathbf{e}_2)$.

Solution. The hint is the whole exercise. Under the dot product, $\mathbf{e}_2$ is already orthogonal to $\mathrm{span}[\mathbf{e}_1,\mathbf{e}_3]$, so its projection would be $\mathbf{0}$ and the distance would be $1$. Under this inner product it is not, because $A_{12} = 1 \neq 0$ and $A_{32} = -1 \neq 0$: the coordinate axes are not perpendicular in this geometry.

(a) The normal equation with the inner product’s matrix inserted in every slot — see the last pitfall on the Orthogonal Projections page — is

$$\mathbf{B}^\top\mathbf{A}\mathbf{B}\,\boldsymbol{\lambda} = \mathbf{B}^\top\mathbf{A}\mathbf{e}_2, \qquad \mathbf{B} = [\mathbf{e}_1\ \ \mathbf{e}_3] .$$

Because $\mathbf{B}$ selects rows and columns $1$ and $3$, both sides read straight off $\mathbf{A}$:

$$\mathbf{B}^\top\mathbf{A}\mathbf{B} = \begin{bmatrix}A_{11} & A_{13}\\ A_{31} & A_{33}\end{bmatrix} = \begin{bmatrix}2&0\\0&2\end{bmatrix}, \qquad \mathbf{B}^\top\mathbf{A}\mathbf{e}_2 = \begin{bmatrix}A_{12}\\ A_{32}\end{bmatrix} = \begin{bmatrix}1\\-1\end{bmatrix}$$

The Gram matrix is diagonal, so no solving is needed:

$$\boldsymbol{\lambda} = \begin{bmatrix}1/2\\ -1/2\end{bmatrix}, \qquad \pi_U(\mathbf{e}_2) = \tfrac12\mathbf{e}_1 - \tfrac12\mathbf{e}_3 = \begin{bmatrix}0.5\\ 0\\ -0.5\end{bmatrix}$$

Not the zero vector. Under this inner product $\mathbf{e}_2$ has a genuine component along $\mathbf{e}_1$ and along $\mathbf{e}_3$.

(b) The residual is

$$\mathbf{e}_2 - \pi_U(\mathbf{e}_2) = \begin{bmatrix}-0.5\\ 1\\ 0.5\end{bmatrix}$$

and the distance must be measured with the same inner product:

$$\lVert\mathbf{r}\rVert^2_{\mathbf{A}} = \mathbf{r}^\top\mathbf{A}\mathbf{r}$$

Compute $\mathbf{A}\mathbf{r}$ first:

row	working	result
1	$2(-0.5) + 1(1) + 0(0.5)$	$0$
2	$1(-0.5) + 2(1) + (-1)(0.5)$	$1$
3	$0(-0.5) + (-1)(1) + 2(0.5)$	$0$

Then $\mathbf{r}\cdot(0,1,0) = 1$, so

$$d(\mathbf{e}_2, U) = \sqrt{1} = 1 .$$

Exactly $1$. And note the trap closing: the Euclidean length of that same residual is $\sqrt{0.25 + 1 + 0.25} = \sqrt{1.5} \approx 1.224745$. Using the dot product to measure a distance in a non-Euclidean geometry gives $1.224745$; the correct answer is $1$. Orthogonality was defined through the inner product, and so is length.

The orthogonality check, in the right inner product: $\mathbf{e}_1^\top\mathbf{A}\mathbf{r} = 0$ and $\mathbf{e}_3^\top\mathbf{A}\mathbf{r} = 0$, both exactly.

(c) The picture: three unit coordinate arrows, the plane $U$ spanned by the first and third, and $\pi_U(\mathbf{e}_2)$ sitting in that plane at $(0.5, 0, -0.5)$ — pointing along the $\mathbf{e}_1 - \mathbf{e}_3$ diagonal, not at the origin. The dashed perpendicular from $\mathbf{e}_2$ down to it looks oblique in Euclidean eyes and is perpendicular in this inner product’s eyes, which is exactly what makes the exercise worth drawing. ∎

ex_3_6.py

import numpy as np
 
A = np.array([[2.0, 1.0, 0.0], [1.0, 2.0, -1.0], [0.0, -1.0, 2.0]])
B = np.array([[1.0, 0.0], [0.0, 0.0], [0.0, 1.0]])      # columns e1 and e3
e2 = np.array([0.0, 1.0, 0.0])
 
G = B.T @ A @ B
c = B.T @ A @ e2
print("B^T A B =", G.tolist(), "  B^T A e2 =", c)
 
lam = np.linalg.solve(G, c)
proj = B @ lam
r = e2 - proj
print("(a) lambda =", lam, " pi_U(e2) =", proj)
print("    A r    =", A @ r)
print("(b) A-squared-norm =", float(r @ A @ r), " distance =", round(float(np.sqrt(r @ A @ r)), 6))
print("    Euclidean norm of the same residual =", round(float(np.linalg.norm(r)), 6),
      "  <- the WRONG answer")
print("orthogonal under A?", float(np.array([1.0, 0, 0]) @ A @ r), float(np.array([0, 0, 1.0]) @ A @ r))
print("A eigenvalues:", np.round(np.linalg.eigvalsh(A), 6))

output

B^T A B = [[2.0, 0.0], [0.0, 2.0]]   B^T A e2 = [ 1. -1.]
(a) lambda = [ 0.5 -0.5]  pi_U(e2) = [ 0.5  0.  -0.5]
    A r    = [0. 1. 0.]
(b) A-squared-norm = 1.0  distance = 1.0
    Euclidean norm of the same residual = 1.224745   <- the WRONG answer
orthogonal under A? 0.0 0.0
A eigenvalues: [0.585786 2.         3.414214]

3.7 — Projections, abstractly

Let $V$ be a vector space and $\pi$ an endomorphism of $V$.

(a) Prove that $\pi$ is a projection if and only if $\mathrm{id}_V - \pi$ is a projection, where $\mathrm{id}_V$ is the identity endomorphism on $V$. (b) Assume now that $\pi$ is a projection. Calculate $\mathrm{Im}(\mathrm{id}_V - \pi)$ and $\ker(\mathrm{id}_V - \pi)$ as a function of $\mathrm{Im}(\pi)$ and $\ker(\pi)$.

Solution (a). Write $\rho := \mathrm{id}_V - \pi$ and expand $\rho^2$:

$$\rho^2 = (\mathrm{id} - \pi)(\mathrm{id} - \pi) = \mathrm{id} - \pi - \pi + \pi^2 = \mathrm{id} - 2\pi + \pi^2$$

So

$$\rho^2 - \rho = (\mathrm{id} - 2\pi + \pi^2) - (\mathrm{id} - \pi) = \pi^2 - \pi .$$

The two differences are the same object. Hence $\rho^2 = \rho$ if and only if $\pi^2 = \pi$, which is the statement in both directions at once. ∎

The one-line version: idempotence of $\pi$ and of $\mathrm{id}-\pi$ are literally the same equation $\pi^2 = \pi$, rearranged.

Solution (b). Two claims.

$\mathrm{Im}(\mathrm{id}-\pi) = \ker(\pi)$.

$\subseteq$: take $\mathbf{v} - \pi(\mathbf{v})$ in the image. Then $\pi(\mathbf{v} - \pi(\mathbf{v})) = \pi(\mathbf{v}) - \pi^2(\mathbf{v}) = \pi(\mathbf{v}) - \pi(\mathbf{v}) = \mathbf{0}$, so it is in $\ker(\pi)$.

$\supseteq$: take $\mathbf{w} \in \ker(\pi)$, so $\pi(\mathbf{w}) = \mathbf{0}$. Then $(\mathrm{id}-\pi)(\mathbf{w}) = \mathbf{w} - \mathbf{0} = \mathbf{w}$, so $\mathbf{w}$ is in the image. ∎

$\ker(\mathrm{id}-\pi) = \mathrm{Im}(\pi)$.

$\subseteq$: if $(\mathrm{id}-\pi)(\mathbf{v}) = \mathbf{0}$ then $\mathbf{v} = \pi(\mathbf{v})$, which exhibits $\mathbf{v}$ as something in the image of $\pi$.

$\supseteq$: take $\pi(\mathbf{u})$ in the image. Then $(\mathrm{id}-\pi)(\pi(\mathbf{u})) = \pi(\mathbf{u}) - \pi^2(\mathbf{u}) = \mathbf{0}$. ∎

Both directions used $\pi^2 = \pi$ and nothing else. So the two maps swap image and kernel:

$$\mathrm{Im}(\mathrm{id}-\pi) = \ker(\pi), \qquad \ker(\mathrm{id}-\pi) = \mathrm{Im}(\pi)$$

which is why $\mathbf{I} - \mathbf{P}$ is the residual operator of §3.6: it keeps exactly what $\mathbf{P}$ throws away, and throws away exactly what $\mathbf{P}$ keeps. Note that nothing here assumed orthogonality — the result holds for oblique projections too, and in that case the two subspaces are complementary without being orthogonal complements.

ex_3_7.py

import numpy as np
 
def check(P, label):
    n = P.shape[0]
    R = np.eye(n) - P
    print(f"{label}")
    print("  P^2 - P  =", f"{np.linalg.norm(P @ P - P):.2e}",
          "   R^2 - R =", f"{np.linalg.norm(R @ R - R):.2e}")
    print("  rank P =", np.linalg.matrix_rank(P), " rank R =", np.linalg.matrix_rank(R),
          " sum =", np.linalg.matrix_rank(P) + np.linalg.matrix_rank(R), " n =", n)
    # Im(R) = ker(P): R's columns are killed by P, and P's columns are killed by R.
    print("  P @ R  =", f"{np.abs(P @ R).max():.2e}", "  R @ P =", f"{np.abs(R @ P).max():.2e}")
 
rng = np.random.default_rng(3)
B = rng.normal(size=(7, 3))
check(B @ np.linalg.inv(B.T @ B) @ B.T, "orthogonal projection onto a 3-dim subspace of R^7")
check(np.array([[1.0, 1.0], [0.0, 0.0]]), "an OBLIQUE projection (not symmetric)")

output

orthogonal projection onto a 3-dim subspace of R^7
  P^2 - P  = 2.41e-16    R^2 - R = 3.11e-16
  rank P = 3  rank R = 4  sum = 7  n = 7
  P @ R  = 1.83e-16   R @ P = 1.83e-16
an OBLIQUE projection (not symmetric)
  P^2 - P  = 0.00e+00    R^2 - R = 0.00e+00
  rank P = 1  rank R = 1  sum = 2  n = 2
  P @ R  = 0.00e+00   R @ P = 0.00e+00

Read the two idempotence columns together. Part (a) says $\rho^2 - \rho$ and $\pi^2 - \pi$ are the same expression, so exact arithmetic would give identical numbers — and the exact case, the oblique projection with integer entries, does: both print $0.00\mathrm{e}{+}00$. In the floating-point case they come out at $2.41\times10^{-16}$ and $3.11\times10^{-16}$, differing because $\mathbf{P}$ and $\mathbf{I}-\mathbf{P}$ are formed by different sequences of operations even though they are algebraically related. Same order of magnitude, both at machine precision; the algebra is exact and the arithmetic is not.

Part (b) is the last line of each block: $\mathbf{P}\mathbf{R} = \mathbf{R}\mathbf{P} = \mathbf{0}$, so each map annihilates the other’s image — which is precisely $\mathrm{Im}(\mathbf{R}) = \ker(\mathbf{P})$ and $\mathrm{Im}(\mathbf{P}) = \ker(\mathbf{R})$. And the ranks add to $n$ in both cases, including the oblique one.

3.8 — Gram-Schmidt

Using the Gram-Schmidt method, turn the basis $B = (\mathbf{b}_1, \mathbf{b}_2)$ of a two-dimensional subspace $U \subseteq \mathbb{R}^3$ into an ONB $C = (\mathbf{c}_1, \mathbf{c}_2)$ of $U$, where

\qquad \mathbf{b}_2 := \begin{bmatrix}-1\\2\\0\end{bmatrix}.$$

Solution.

Step 1. $\mathbf{u}_1 := \mathbf{b}_1 = (1,1,1)^\top$, kept unchanged.

Step 2. Subtract the projection of $\mathbf{b}_2$ onto $\mathbf{u}_1$. The coefficient is

$$\frac{\langle\mathbf{u}_1,\mathbf{b}_2\rangle}{\langle\mathbf{u}_1,\mathbf{u}_1\rangle} = \frac{-1 + 2 + 0}{1 + 1 + 1} = \frac{1}{3}$$ $$\mathbf{u}_2 = \mathbf{b}_2 - \tfrac13\mathbf{u}_1 = \begin{bmatrix}-1\\2\\0\end{bmatrix} - \tfrac13\begin{bmatrix}1\\1\\1\end{bmatrix} = \begin{bmatrix}-4/3\\ 5/3\\ -1/3\end{bmatrix}$$

Check: $\langle\mathbf{u}_1,\mathbf{u}_2\rangle = -\tfrac43 + \tfrac53 - \tfrac13 = 0$ ✓

Step 3. Normalise. $\lVert\mathbf{u}_1\rVert = \sqrt3$, and

$$\lVert\mathbf{u}_2\rVert = \tfrac13\sqrt{16 + 25 + 1} = \frac{\sqrt{42}}{3} \approx 2.160247$$

so the $\tfrac13$ cancels and

$$\mathbf{c}_1 = \frac{1}{\sqrt3}\begin{bmatrix}1\\1\\1\end{bmatrix}, \qquad \mathbf{c}_2 = \frac{1}{\sqrt{42}}\begin{bmatrix}-4\\5\\-1\end{bmatrix}$$ $$\mathbf{C}^\top\mathbf{C} = \mathbf{I}_2$$

∎

ex_3_8.py

import numpy as np
 
b1 = np.array([1.0, 1.0, 1.0])
b2 = np.array([-1.0, 2.0, 0.0])
 
u1 = b1.copy()
coeff = float(u1 @ b2) / float(u1 @ u1)
u2 = b2 - coeff * u1
print("coefficient  =", coeff, "  (should be 1/3)")
print("u2           =", u2)
print("<u1, u2>     =", float(u1 @ u2))
 
c1, c2 = u1 / np.linalg.norm(u1), u2 / np.linalg.norm(u2)
print("||u1|| =", np.linalg.norm(u1), " sqrt(3) =", np.sqrt(3))
print("||u2|| =", np.linalg.norm(u2), " sqrt(42)/3 =", np.sqrt(42) / 3)
print("c2 * sqrt(42) =", np.round(c2 * np.sqrt(42), 6))
C = np.stack([c1, c2], axis=1)
print("C^T C =")
print(np.round(C.T @ C, 15))

output

coefficient  = 0.3333333333333333   (should be 1/3)
u2           = [-1.33333333  1.66666667 -0.33333333]
<u1, u2>     = 1.6653345369377348e-16
||u1|| = 1.7320508075688772  sqrt(3) = 1.7320508075688772
||u2|| = 2.1602468994692865  sqrt(42)/3 = 2.160246899469287
c2 * sqrt(42) = [-4.  5. -1.]
C^T C =
[[1. 0.]
 [0. 1.]]

3.9 — Two inequalities from Cauchy-Schwarz

Let $n \in \mathbb{N}^*$ and let $x_1, \dots, x_n > 0$ be $n$ positive real numbers so that $x_1 + \dots + x_n = 1$. Use the Cauchy-Schwarz inequality and show that (a) $\displaystyle\sum_{i=1}^n x_i^2 \geq \frac{1}{n}$ (b) $\displaystyle\sum_{i=1}^n \frac{1}{x_i} \geq n^2$ Hint: Think about the dot product on $\mathbb{R}^n$. Then, choose specific vectors $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$ and apply the Cauchy-Schwarz inequality.

Solution. Both parts are the same trick — choose the two vectors so that the inner product is the thing you know and the norms are the things you want.

(a) Take $\mathbf{x} = (x_1, \dots, x_n)$ and $\mathbf{y} = (1, 1, \dots, 1)$. Then

$$\langle\mathbf{x},\mathbf{y}\rangle = \sum_i x_i = 1, \qquad \lVert\mathbf{x}\rVert^2 = \sum_i x_i^2, \qquad \lVert\mathbf{y}\rVert^2 = n$$

Cauchy-Schwarz squared says $\langle\mathbf{x},\mathbf{y}\rangle^2 \leq \lVert\mathbf{x}\rVert^2\lVert\mathbf{y}\rVert^2$, so

$$1 \leq \left(\sum_i x_i^2\right) n \qquad\Longrightarrow\qquad \sum_i x_i^2 \geq \frac{1}{n} .$$

∎

(b) Now take $\mathbf{u} = (\sqrt{x_1}, \dots, \sqrt{x_n})$ and $\mathbf{v} = \left(\tfrac{1}{\sqrt{x_1}}, \dots, \tfrac{1}{\sqrt{x_n}}\right)$ — legal because every $x_i > 0$. Then

$$\langle\mathbf{u},\mathbf{v}\rangle = \sum_i \sqrt{x_i}\cdot\frac{1}{\sqrt{x_i}} = \sum_i 1 = n, \qquad \lVert\mathbf{u}\rVert^2 = \sum_i x_i = 1, \qquad \lVert\mathbf{v}\rVert^2 = \sum_i \frac{1}{x_i}$$

Cauchy-Schwarz squared:

$$n^2 \leq 1 \cdot \sum_i \frac{1}{x_i} \qquad\Longrightarrow\qquad \sum_i \frac{1}{x_i} \geq n^2 .$$

∎

When is each an equality? Cauchy-Schwarz is tight exactly when the two vectors are parallel. In (a) that means $\mathbf{x}$ parallel to $(1,\dots,1)$, so every $x_i$ equal — and with the sum fixed at $1$, $x_i = 1/n$. In (b) it means $\sqrt{x_i}$ proportional to $1/\sqrt{x_i}$, so again all $x_i$ equal.

Both bounds are attained at the uniform distribution and nowhere else. Which makes (a) a statement you have seen elsewhere: $\sum x_i^2$ is minimised by the uniform distribution, so it measures concentration. It is the Simpson index, and $1 - \sum x_i^2$ is Gini impurity — the split criterion in a decision tree.

ex_3_9.py

import numpy as np
 
rng = np.random.default_rng(7)
worst_a, worst_b = np.inf, np.inf
for _ in range(200000):
    n = int(rng.integers(2, 9))
    v = rng.random(n)
    v = v / v.sum()                              # positive, summing to 1
    worst_a = min(worst_a, float(np.sum(v ** 2) * n))       # >= 1 iff (a) holds
    worst_b = min(worst_b, float(np.sum(1 / v) / n ** 2))   # >= 1 iff (b) holds
 
print("200000 random distributions")
print("  smallest n * sum(x^2)     :", round(worst_a, 6), " (the bound is 1)")
print("  smallest sum(1/x) / n^2   :", round(worst_b, 6), " (the bound is 1)")
print()
for n in (2, 3, 5, 10):
    v = np.full(n, 1.0 / n)
    print(f"  n={n:2}  uniform: sum x^2 = {float(np.sum(v ** 2)):.6f} = 1/n = {1 / n:.6f}"
          f"   sum 1/x = {float(np.sum(1 / v)):.1f} = n^2 = {n ** 2}")

output

200000 random distributions
  smallest n * sum(x^2)     : 1.0  (the bound is 1)
  smallest sum(1/x) / n^2   : 1.0  (the bound is 1)
 
  n= 2  uniform: sum x^2 = 0.500000 = 1/n = 0.500000   sum 1/x = 4.0 = n^2 = 4
  n= 3  uniform: sum x^2 = 0.333333 = 1/n = 0.333333   sum 1/x = 9.0 = n^2 = 9
  n= 5  uniform: sum x^2 = 0.200000 = 1/n = 0.200000   sum 1/x = 25.0 = n^2 = 25
  n= 10 uniform: sum x^2 = 0.100000 = 1/n = 0.100000   sum 1/x = 100.0 = n^2 = 100

Both minima come out at exactly $1.0$ over two hundred thousand samples — that is the bound being touched, and it happens because the sampler occasionally draws something very close to uniform. Neither ratio ever dips below $1$.

3.10 — Rotate two vectors

Rotate the vectors $\mathbf{x}_1 := \begin{bmatrix}2\\3\end{bmatrix}$, $\mathbf{x}_2 := \begin{bmatrix}0\\-1\end{bmatrix}$ by $30°$.

Solution. $\cos 30° = \tfrac{\sqrt3}{2}$ and $\sin 30° = \tfrac12$, so by Equation 3.76

$$\mathbf{R}(30°) = \begin{bmatrix}\tfrac{\sqrt3}{2} & -\tfrac12\\[3pt] \tfrac12 & \tfrac{\sqrt3}{2}\end{bmatrix}$$ $$\mathbf{R}\mathbf{x}_1 = \begin{bmatrix}2\cdot\tfrac{\sqrt3}{2} - 3\cdot\tfrac12\\[3pt] 2\cdot\tfrac12 + 3\cdot\tfrac{\sqrt3}{2}\end{bmatrix} = \begin{bmatrix}\sqrt3 - \tfrac32\\[3pt] 1 + \tfrac{3\sqrt3}{2}\end{bmatrix} \approx \begin{bmatrix}0.232051\\ 3.598076\end{bmatrix}$$ $$\mathbf{R}\mathbf{x}_2 = \begin{bmatrix}0 + \tfrac12\\[3pt] 0 - \tfrac{\sqrt3}{2}\end{bmatrix} = \begin{bmatrix}\tfrac12\\[3pt] -\tfrac{\sqrt3}{2}\end{bmatrix} \approx \begin{bmatrix}0.5\\ -0.866025\end{bmatrix}$$

$\mathbf{x}_2 = -\mathbf{e}_2$, so $\mathbf{R}\mathbf{x}_2$ is minus the second column of $\mathbf{R}$ — rotating a basis vector reads off a column, no arithmetic required.

Checks. $\lVert\mathbf{x}_1\rVert = \sqrt{13} = 3.605551$ before and after. $\lVert\mathbf{x}_2\rVert = 1$ before and after. The angle between them is $146.309932°$ before and $146.309932°$ after — nine matching digits. $\det\mathbf{R} = 1.000000000000000$. ∎

ex_3_10.py

import numpy as np
 
def rot2(theta):
    c, s = np.cos(theta), np.sin(theta)
    return np.array([[c, -s], [s, c]])
 
R = rot2(np.radians(30.0))
x1, x2 = np.array([2.0, 3.0]), np.array([0.0, -1.0])
y1, y2 = R @ x1, R @ x2
 
print("R(30) =", np.round(R, 6).tolist())
print("x1 ->", np.round(y1, 6), " exact:", round(np.sqrt(3) - 1.5, 6), round(1 + 1.5 * np.sqrt(3), 6))
print("x2 ->", np.round(y2, 6), " exact:", 0.5, round(-np.sqrt(3) / 2, 6))
 
cos = lambda a, b: float(a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
print("||x1||:", round(float(np.linalg.norm(x1)), 6), "->", round(float(np.linalg.norm(y1)), 6))
print("||x2||:", round(float(np.linalg.norm(x2)), 6), "->", round(float(np.linalg.norm(y2)), 6))
print("angle: ", round(float(np.degrees(np.arccos(np.clip(cos(x1, x2), -1, 1)))), 6),
      "->", round(float(np.degrees(np.arccos(np.clip(cos(y1, y2), -1, 1)))), 6))
print("det R =", f"{np.linalg.det(R):.15f}")

output

R(30) = [[0.866025, -0.5], [0.5, 0.866025]]
x1 -> [0.232051 3.598076]  exact: 0.232051 3.598076
x2 -> [ 0.5      -0.866025]  exact: 0.5 -0.866025
||x1||: 3.605551 -> 3.605551
||x2||: 1.0 -> 1.0
angle:  146.309932 -> 146.309932
det R = 1.000000000000000

What the exercises were actually testing

#	looks like it tests	actually tests
3.1	reading a formula as a matrix	that positive definiteness can be shown by completing the square, without eigenvalues
3.2	positive definiteness	symmetry — the quadratic form here is positive definite and the map is still not an inner product
3.3	two distance computations	that both answers are correct, and differ by $46\%$
3.4	two angle computations	the same, for angles
3.5	the projection formula	that a spanning set is not a basis — the rank is 3, not 4, and Eq 3.59 is undefined as posed
3.6	projection onto a coordinate plane	that orthogonality and length both depend on the inner product — under the dot product the answer would be $\mathbf{0}$ and distance $1$; here it is $(0.5, 0, -0.5)$ and distance $1$, while the Euclidean length of the residual is $1.224745$
3.7	an abstract proof	that $\mathbf{I}-\mathbf{P}$ swaps image and kernel, for oblique projections too
3.8	Gram-Schmidt arithmetic	that the awkward thirds cancel against the norm
3.9	Cauchy-Schwarz	choosing the two vectors so the inner product is what you know
3.10	matrix-vector multiplication	that rotating $\mathbf{e}_i$ reads off column $i$

Three of the ten (3.2, 3.5, 3.6) fail if you apply the formula without checking its preconditions. That ratio is not an accident.

pch.quizTag Did the traps catch you?

1. Exercise 3.2's matrix [[2, 0], [1, 2]] fails to be an inner product. Which condition fails, and what is the shortest witness?

   A Positive definiteness; the witness is (1, -1) B Symmetry; the witness is <e1, e2> = 0 against <e2, e1> = 1 C Bilinearity; matrix products are not bilinear D Nothing fails — it is an inner product
   pch.quizShowAnswer

   B — Symmetry; the witness is <e1, e2> = 0 against <e2, e1> = 1 — The quadratic form 2x1^2 + x1x2 + 2x2^2 is positive definite: only the symmetric part of A affects it, and that part has eigenvalues 1.5 and 2.5. Testing definiteness alone passes this matrix, which is the trap.

2. Exercise 3.5 gives four spanning vectors in R^5. What must you do first, and why?

   A Normalise them B Check the rank — it is 3, so they are not a basis, B-transpose B is singular with condition number 2e17, and Equation 3.59 is undefined as posed. The fourth vector equals u1 + 2 u2 + u3 C Sort them by length D Nothing; the formula handles it
   pch.quizShowAnswer

   B — Check the rank — it is 3, so they are not a basis, B-transpose B is singular with condition number 2e17, and Equation 3.59 is undefined as posed. The fourth vector equals u1 + 2 u2 + u3 — The book flags exactly this in a margin note in section 3.8.2. np.linalg.lstsq happens to return the right projection anyway, because it uses a minimum-norm pseudo-inverse rather than an inverse — but the formula as written does not.

3. In Exercise 3.6, what is d(e2, U) and what is the tempting wrong answer?

   A Exactly 1; the wrong answer is 0, from assuming e2 is already orthogonal to span(e1, e3) B Exactly 1; the wrong answer is 1.224745, from measuring the residual with the dot product instead of the exercise's inner product C 1.224745; the wrong answer is 1 D Zero, since e2 is a basis vector
   pch.quizShowAnswer

   B — Exactly 1; the wrong answer is 1.224745, from measuring the residual with the dot product instead of the exercise's inner product — Both wrong answers are available. Assuming orthogonality gives a projection of 0; measuring the correct residual (-0.5, 1, 0.5) with the dot product gives sqrt(1.5) = 1.224745. The correct A-norm of that residual is exactly 1.

4. Exercise 3.9's two bounds are attained at the same distribution. Which, and what does part (a) turn out to be?

   A At the most concentrated distribution; it is the entropy B At the uniform distribution, since Cauchy-Schwarz is tight exactly when the two vectors are parallel — and sum of x-i squared is the Simpson index, whose complement is Gini impurity C At no distribution; the bounds are strict D At any distribution; both hold with equality always
   pch.quizShowAnswer

   B — At the uniform distribution, since Cauchy-Schwarz is tight exactly when the two vectors are parallel — and sum of x-i squared is the Simpson index, whose complement is Gini impurity — In part (a) tightness needs x parallel to the all-ones vector, so every x-i equal, hence 1/n each. In part (b) it needs sqrt(x-i) proportional to 1 over sqrt(x-i), which again forces all equal.

🧪 Try It Yourself

Exercise A – Spot the non-inner-product

pch.exerciseTag pch.exerciseTitle

Exercise B – Exercise 3.5 the wrong way, then the right way

pch.exerciseTag pch.exerciseTitle

Exercise C – Exercise 3.6, both answers

pch.exerciseTag pch.exerciseTitle

Exercise D – Exercise 3.7, measured

pch.exerciseTag pch.exerciseTitle

Exercise E – Exercise 3.9’s equality case

pch.exerciseTag pch.exerciseTitle

Recall card

• Exercise 3.2 is about symmetry, not definiteness — the quadratic form of a non-symmetric matrix can be positive definite while the map fails to be an inner product, because only the symmetric part affects the form.
• Exercise 3.5’s spanning set has rank three, not four, so the Gram matrix is singular and the projection formula is undefined as posed. Determine a basis first.
• Exercise 3.6’s coordinate axes are not perpendicular in its inner product, so the projection of the second basis vector is not zero, and the distance must be measured with the same inner product — one, not the Euclidean 1.224745.
• Exercise 3.7: the residual operator swaps image and kernel. The image of the identity minus pi is the kernel of pi and vice versa, and both idempotence conditions are the same equation.
• Exercise 3.9’s trick is choosing the vectors so that the inner product is the quantity you already know, and both bounds are tight only at the uniform distribution.
• Both distance exercises and both angle exercises give two different correct answers, which is the chapter’s central point restated as arithmetic.

Next: Chapter 3 Formula Sheet — every result on one page.