Chapter 2 Exercises and Solutions

These are the exercises from the end of Chapter 2 of Mathematics for Machine Learning (page 64 of the December 2019 draft), restated in this module’s notation. The book provides them “mostly to be done by pen and paper”, and that is the right way to use this page.

Work each one before opening its solution. Every solution is followed by a NumPy block that verifies the answer, so you can also check your own working against something other than my arithmetic.

How to use this page

1. Read the problem. Close the disclosure.
2. Do it on paper.
3. Open the solution and compare methods, not just answers — several of these have many correct-looking forms.
4. Run the verification block if your answer differs from mine, to find out which of us is wrong.

The last point is not a joke. Several of these exercises have solutions that look nothing alike and are both correct, because neither a particular solution nor a basis is unique (§2.3).

§2.4 — Groups

Exercise 2.1

Consider $(\mathbb{R}\setminus\{-1\}, \star)$ where

$$a \star b := ab + a + b, \qquad a, b \in \mathbb{R}\setminus\{-1\}$$

(a) Show that $(\mathbb{R}\setminus\{-1\}, \star)$ is an Abelian group. (b) Solve $3 \star x \star x = 15$ in this group.

Solution

The key observation, and everything follows from it:

$$a \star b = ab + a + b = (a+1)(b+1) - 1$$

So $\star$ is ordinary multiplication in disguise, shifted by one. Define $\varphi(a) = a + 1$. Then $\varphi(a \star b) = \varphi(a)\varphi(b)$, and $\varphi$ is a bijection from $\mathbb{R}\setminus\{-1\}$ onto $\mathbb{R}\setminus\{0\}$. Since $(\mathbb{R}\setminus\{0\}, \cdot)$ is a known Abelian group (§2.4’s Example 2.10), so is this one — and the excluded value $-1$ is excluded precisely because $\varphi(-1) = 0$, the one element multiplication has to lose.

(a) Checking the four axioms directly:

• Closure. $(a+1)(b+1) - 1 = -1$ would need $(a+1)(b+1) = 0$, so $a = -1$ or $b = -1$ — both excluded. So $a \star b \neq -1$ and the result stays in the set. ✓
• Associativity. $(a \star b) \star c = (a+1)(b+1)(c+1) - 1$, symmetric in all three, so it equals $a \star (b \star c)$. ✓
• Neutral element. Need $(a+1)(e+1) = a+1$, so $e + 1 = 1$ and $\mathbf{e = 0}$. Check: $a \star 0 = 0 + a + 0 = a$ ✓
• Inverses. Need $(a+1)(b+1) = 1$, so $b = \frac{1}{a+1} - 1 = \boxed{-\dfrac{a}{a+1}}$, which is defined for every $a \neq -1$. ✓
• Abelian. $(a+1)(b+1) = (b+1)(a+1)$. ✓

(b) Using the product form:

$$3 \star x \star x = (3+1)(x+1)(x+1) - 1 = 4(x+1)^2 - 1 = 15$$$$4(x+1)^2 = 16 \;\Longrightarrow\; (x+1)^2 = 4 \;\Longrightarrow\; x + 1 = \pm2$$

so $\boxed{x = 1 \text{ or } x = -3}$. Both are in $\mathbb{R}\setminus\{-1\}$, so both are valid.

Where people get stuck: taking only the positive square root and reporting $x = 1$. The group is multiplicative in disguise and $(-2)^2 = 4$ just as well as $2^2$ does.

verify_2_1.py

import numpy as np
 
star = lambda a, b: a * b + a + b
 
# The product identity that makes the whole exercise easy.
print("a*b == (a+1)(b+1)-1 :",
      all(np.isclose(star(a, b), (a + 1) * (b + 1) - 1)
          for a, b in [(2, 3), (-0.5, 4), (7, -0.9)]))
 
# Neutral element is 0, not 1.
print("neutral element 0   :", [star(a, 0.0) for a in (2.0, -0.5, 7.0)])
 
# The inverse formula.
inv = lambda a: -a / (a + 1)
print("a * inv(a) == 0     :", [round(star(a, inv(a)), 12) for a in (2.0, -0.5, 7.0)])
 
# Both roots of part (b).
for x in (1.0, -3.0):
    print(f"3 * x * x with x = {x:+.0f} -> {star(star(3.0, x), x)}")

text

a*b == (a+1)(b+1)-1 : True
neutral element 0   : [2.0, -0.5, 7.0]
a * inv(a) == 0     : [0.0, 0.0, 0.0]
3 * x * x with x = +1 -> 15.0
3 * x * x with x = -3 -> 15.0

Exercise 2.3

Consider the set of $3\times3$ matrices

$$\mathcal{G} = \left\{\begin{bmatrix}1 & x & z\\ 0 & 1 & y\\ 0 & 0 & 1\end{bmatrix} \in \mathbb{R}^{3\times3} \;\middle|\; x, y, z \in \mathbb{R}\right\}$$

with standard matrix multiplication. Is $(\mathcal{G}, \cdot)$ a group? Is it Abelian?

Solution

It is a group, and it is not Abelian.

Write $g(x,y,z)$ for the matrix with those three entries. Multiplying two of them:

$$g(x_1,y_1,z_1)\,g(x_2,y_2,z_2) = g(x_1+x_2,\; y_1+y_2,\; z_1+z_2+x_1y_2)$$

That single formula settles almost everything. Working it out entry by entry, the $(1,3)$ entry is $1\cdot z_2 + x_1 y_2 + z_1 \cdot 1 = z_1 + z_2 + x_1y_2$, and the note to take from it is the asymmetric $x_1y_2$ term.

• Closure. The product has $1$s on the diagonal and $0$s below, so it is in $\mathcal{G}$. ✓
• Associativity. Inherited from matrix multiplication, which is always associative (§2.2). ✓
• Neutral element. $g(0,0,0) = \mathbf{I}_3 \in \mathcal{G}$. ✓
• Inverses. Solve $g(x,y,z)\,g(x',y',z') = g(0,0,0)$: we need $x' = -x$, $y' = -y$, and $z + z' + xy' = 0$, giving $z' = -z + xy$. So $g(x,y,z)^{-1} = g(-x,\,-y,\,xy - z)$ which exists for every $x,y,z$. ✓

So $(\mathcal{G}, \cdot)$ is a group.

Not Abelian, and the $x_1y_2$ term is why. Swapping the factors gives $z_1 + z_2 + x_2y_1$ instead, so the two products agree only when $x_1y_2 = x_2y_1$. A single counterexample suffices:

$$g(1,2,3)\,g(4,5,6) = g(5,7,14), \qquad g(4,5,6)\,g(1,2,3) = g(5,7,17)$$

$14 \neq 17$. Note the $x$ and $y$ entries agree in both orders — only the $z$ entry differs, which is exactly where the asymmetric term lives.

Aside worth knowing: this is the Heisenberg group, and $\mathcal{G} \subset GL(3,\mathbb{R})$ (§2.4’s Definition 2.8) — a subgroup of the general linear group.

verify_2_3.py

import numpy as np
 
def g(x, y, z):
    return np.array([[1.0, x, z], [0.0, 1.0, y], [0.0, 0.0, 1.0]])
 
A, B = g(1, 2, 3), g(4, 5, 6)
 
def in_G(M):
    return (np.allclose(np.diag(M), 1) and np.isclose(M[1, 0], 0)
            and np.isclose(M[2, 0], 0) and np.isclose(M[2, 1], 0))
 
print("closure  :", in_G(A @ B))
print("A@B = g(%g, %g, %g)" % ((A @ B)[0, 1], (A @ B)[1, 2], (A @ B)[0, 2]))
print("B@A = g(%g, %g, %g)" % ((B @ A)[0, 1], (B @ A)[1, 2], (B @ A)[0, 2]))
print("Abelian  :", np.allclose(A @ B, B @ A))
print("inverse formula g(-x,-y,xy-z):",
      np.allclose(g(1, 2, 3) @ g(-1, -2, 1 * 2 - 3), np.eye(3)))
print("also from the left            :",
      np.allclose(g(-1, -2, 1 * 2 - 3) @ g(1, 2, 3), np.eye(3)))

text

closure  : True
A@B = g(5, 7, 14)
B@A = g(5, 7, 17)
Abelian  : False
inverse formula g(-x,-y,xy-z): True
also from the left            : True

§2.2 — Matrix products

Exercise 2.4

Compute the following matrix products, if possible.

(a) $\begin{bmatrix}1&2\\4&5\\7&8\end{bmatrix}\begin{bmatrix}1&1&0\\0&1&1\\1&0&1\end{bmatrix}$   (b) $\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix}\begin{bmatrix}1&1&0\\0&1&1\\1&0&1\end{bmatrix}$   (c) $\begin{bmatrix}1&1&0\\0&1&1\\1&0&1\end{bmatrix}\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix}$

(d) $\begin{bmatrix}1&2&1&2\\4&1&-1&-4\end{bmatrix}\begin{bmatrix}0&3\\1&-1\\2&1\\5&2\end{bmatrix}$   (e) $\begin{bmatrix}0&3\\1&-1\\2&1\\5&2\end{bmatrix}\begin{bmatrix}1&2&1&2\\4&1&-1&-4\end{bmatrix}$

Solution

(a) Not possible. The first matrix is $3\times2$ and the second is $3\times3$. The neighbouring dimensions are $2$ and $3$ — they do not match, so the product is undefined (§2.2, Equation 2.14).

Check the dimensions before computing anything. This is the only one of the five that is a trap.

(b) $3\times3$ times $3\times3$, giving $3\times3$:

$$\begin{bmatrix}4&3&5\\10&9&11\\16&15&17\end{bmatrix}$$

Spot-check the $(1,1)$ entry: row $(1,2,3)$ against column $(1,0,1)$ gives $1 + 0 + 3 = 4$ ✓.

(c) The same two matrices, other order — and a different answer, which is §2.2’s non-commutativity:

$$\begin{bmatrix}5&7&9\\11&13&15\\8&10&12\end{bmatrix}$$

Here both products are defined and have the same shape; they are simply unequal.

(d) $2\times4$ times $4\times2$, giving $2\times2$:

$$\begin{bmatrix}14&6\\-21&2\end{bmatrix}$$

Spot-check $(1,1)$: $(1,2,1,2)\cdot(0,1,2,5) = 0 + 2 + 2 + 10 = 14$ ✓.

(e) The same two, reversed — $4\times2$ times $2\times4$, giving $4\times4$:

$$\begin{bmatrix}12&3&-3&-12\\-3&1&2&6\\6&5&1&0\\13&12&3&2\end{bmatrix}$$

This is the book’s Figure 2.5 point in exercise form: both (d) and (e) are defined, and their results are $2\times2$ and $4\times4$. Not merely unequal — not even comparable.

verify_2_4.py

import numpy as np
 
M1 = np.array([[1., 2], [4, 5], [7, 8]])
M2 = np.array([[1., 1, 0], [0, 1, 1], [1, 0, 1]])
M3 = np.array([[1., 2, 3], [4, 5, 6], [7, 8, 9]])
M4 = np.array([[1., 2, 1, 2], [4, 1, -1, -4]])
M5 = np.array([[0., 3], [1, -1], [2, 1], [5, 2]])
 
print("(a) inner dims", M1.shape[1], "vs", M2.shape[0], "-> defined:", M1.shape[1] == M2.shape[0])
try:
    M1 @ M2
except ValueError as e:
    print("    NumPy agrees:", str(e).split(":")[0])
 
print("(b)\n", M3 @ M2)
print("(c)\n", M2 @ M3)
print("(b) == (c)?", np.allclose(M3 @ M2, M2 @ M3))
print("(d)\n", M4 @ M5, " shape", (M4 @ M5).shape)
print("(e)\n", M5 @ M4, " shape", (M5 @ M4).shape)
print("(d) and (e) even comparable?", (M4 @ M5).shape == (M5 @ M4).shape)

text

(a) inner dims 2 vs 3 -> defined: False
    NumPy agrees: matmul
(b)
 [[ 4.  3.  5.]
 [10.  9. 11.]
 [16. 15. 17.]]
(c)
 [[ 5.  7.  9.]
 [11. 13. 15.]
 [ 8. 10. 12.]]
(b) == (c)? False
(d)
 [[ 14.   6.]
 [-21.   2.]]  shape (2, 2)
(e)
 [[ 12.   3.  -3. -12.]
 [ -3.   1.   2.   6.]
 [  6.   5.   1.   0.]
 [ 13.  12.   3.   2.]]  shape (4, 4)
(d) and (e) even comparable? False

§2.3 — Solving systems

Exercise 2.5

Find the set $S$ of all solutions of $\mathbf{A}\mathbf{x} = \mathbf{b}$.

(a) $\mathbf{A} = \begin{bmatrix}1&1&-1&-1\\2&5&-7&-5\\2&-1&1&3\\5&2&-4&2\end{bmatrix}$, $\mathbf{b} = \begin{bmatrix}1\\-2\\4\\6\end{bmatrix}$   (b) $\mathbf{A} = \begin{bmatrix}1&-1&0&0&1\\1&1&0&-3&0\\2&-1&0&1&-1\\-1&2&0&-2&-1\end{bmatrix}$, $\mathbf{b} = \begin{bmatrix}3\\6\\5\\-1\end{bmatrix}$

Solution

(a) $S = \emptyset$ — no solution.

Reducing the augmented matrix $[\mathbf{A}\mid\mathbf{b}]$ gives

$$\begin{bmatrix} 1 & 0 & 0 & \tfrac23 & 0\\ 0 & 1 & 0 & -\tfrac83 & 0\\ 0 & 0 & 1 & -1 & 0\\ 0 & 0 & 0 & 0 & 1 \end{bmatrix}$$

The last row reads $0 = 1$. So $\operatorname{rk}(\mathbf{A}) = 3$ while $\operatorname{rk}(\mathbf{A}\mid\mathbf{b}) = 4$, and §2.6’s solvability criterion says the system is inconsistent. This is §2.1’s ”$0 =$ nonzero” contradiction, in four variables.

(b) $S$ is a two-dimensional affine subspace of $\mathbb{R}^5$.

The reduced augmented matrix is

$$\begin{bmatrix} 1 & 0 & 0 & 0 & -1 & 3\\ 0 & 1 & 0 & 0 & -2 & 0\\ 0 & 0 & 0 & 1 & -1 & -1\\ 0 & 0 & 0 & 0 & 0 & 0 \end{bmatrix}$$

Pivots in columns 1, 2 and 4, so $x_1, x_2, x_4$ are basic and $x_3, x_5$ are free. Note that column 3 of $\mathbf{A}$ is entirely zero, which is why $x_3$ is free without any work — it appears in no equation at all.

Reading the rows: $x_1 = 3 + x_5$, $x_2 = 2x_5$, $x_4 = -1 + x_5$. So

$$S = \left\{\begin{bmatrix}3\\0\\0\\-1\\0\end{bmatrix} + \lambda_1\begin{bmatrix}0\\0\\1\\0\\0\end{bmatrix} + \lambda_2\begin{bmatrix}1\\2\\0\\1\\1\end{bmatrix},\;\; \lambda_1, \lambda_2 \in \mathbb{R}\right\}$$

Dimension check: $n - \operatorname{rk}(\mathbf{A}) = 5 - 3 = 2$ ✓ — matching the two free variables and the two direction vectors.

verify_2_5.py

import numpy as np
 
def rref(M, tol=1e-10):
    A = M.astype(float).copy(); rows, cols = A.shape; piv, r = [], 0
    for c in range(cols):
        if r >= rows: break
        p = next((i for i in range(r, rows) if abs(A[i, c]) > tol), None)
        if p is None: continue
        A[[r, p]] = A[[p, r]]; A[r] = A[r] / A[r, c]
        for i in range(rows):
            if i != r and abs(A[i, c]) > tol: A[i] = A[i] - A[i, c] * A[r]
        piv.append(c); r += 1
    return A, piv
 
# ---- (a) inconsistent -------------------------------------------------
Aa = np.array([[1., 1, -1, -1], [2, 5, -7, -5], [2, -1, 1, 3], [5, 2, -4, 2]])
ba = np.array([1., -2, 4, 6])
print("(a) rk(A) =", np.linalg.matrix_rank(Aa),
      " rk(A|b) =", np.linalg.matrix_rank(np.c_[Aa, ba]),
      " -> consistent:", np.linalg.matrix_rank(Aa) == np.linalg.matrix_rank(np.c_[Aa, ba]))
 
# ---- (b) two free variables ------------------------------------------
Ab = np.array([[1., -1, 0, 0, 1], [1, 1, 0, -3, 0],
               [2, -1, 0, 1, -1], [-1, 2, 0, -2, -1]])
bb = np.array([3., 6, 5, -1])
R, piv = rref(np.c_[Ab, bb])
print("\n(b) pivots:", piv, " free:", [c for c in range(5) if c not in piv])
print("    rk(A) =", np.linalg.matrix_rank(Ab), " dimension of S:", 5 - np.linalg.matrix_rank(Ab))
 
xp = np.array([3., 0, 0, -1, 0])
d1 = np.array([0., 0, 1, 0, 0])
d2 = np.array([1., 2, 0, 1, 1])
print("    particular works :", np.allclose(Ab @ xp, bb))
print("    d1 in the kernel :", np.allclose(Ab @ d1, 0))
print("    d2 in the kernel :", np.allclose(Ab @ d2, 0))
print("    xp + 3d1 - 2d2 solves:", np.allclose(Ab @ (xp + 3 * d1 - 2 * d2), bb))

text

(a) rk(A) = 3  rk(A|b) = 4  -> consistent: False
 
(b) pivots: [0, 1, 3]  free: [2, 4]
    rk(A) = 3  dimension of S: 2
    particular works : True
    d1 in the kernel : True
    d2 in the kernel : True
    xp + 3d1 - 2d2 solves: True

Exercise 2.6

Using Gaussian elimination, find all solutions of $\mathbf{A}\mathbf{x} = \mathbf{b}$ with

$$\mathbf{A} = \begin{bmatrix}0&1&0&0&1&0\\0&0&0&1&1&0\\0&1&0&0&0&1\end{bmatrix}, \qquad \mathbf{b} = \begin{bmatrix}2\\-1\\1\end{bmatrix}$$

Solution

Reducing $[\mathbf{A}\mid\mathbf{b}]$:

$$\begin{bmatrix} 0 & 1 & 0 & 0 & 0 & 1 & 1\\ 0 & 0 & 0 & 1 & 0 & 1 & -2\\ 0 & 0 & 0 & 0 & 1 & -1 & 1 \end{bmatrix}$$

Pivots in columns 2, 4, 5, so the basic variables are $x_2, x_4, x_5$ and the free ones are $x_1, x_3, x_6$. Note $x_1$ and $x_3$ are free because columns 1 and 3 of $\mathbf{A}$ are entirely zero — those variables appear in no equation.

Reading off: $x_2 = 1 - x_6$, $x_4 = -2 - x_6$, $x_5 = 1 + x_6$. Setting all free variables to zero gives the particular solution $(0, 1, 0, -2, 1, 0)^\top$, and

$$S = \left\{\begin{bmatrix}0\\1\\0\\-2\\1\\0\end{bmatrix} + \lambda_1\begin{bmatrix}1\\0\\0\\0\\0\\0\end{bmatrix} + \lambda_2\begin{bmatrix}0\\0\\1\\0\\0\\0\end{bmatrix} + \lambda_3\begin{bmatrix}0\\-1\\0\\-1\\1\\1\end{bmatrix}\right\}$$

Dimension: $6 - 3 = 3$ ✓, matching the three free variables.

Where people get stuck: forgetting the zero columns. It is tempting to reduce only the “interesting” $3\times3$ block and report a unique solution. The zero columns are real variables, and they are free.

verify_2_6.py

import numpy as np
 
A = np.array([[0., 1, 0, 0, 1, 0],
              [0, 0, 0, 1, 1, 0],
              [0, 1, 0, 0, 0, 1]])
b = np.array([2., -1, 1])
 
print("rk(A) =", np.linalg.matrix_rank(A),
      " rk(A|b) =", np.linalg.matrix_rank(np.c_[A, b]),
      " dim(S) =", 6 - np.linalg.matrix_rank(A))
print("columns 1 and 3 are entirely zero:",
      np.allclose(A[:, 0], 0), np.allclose(A[:, 2], 0))
 
xp = np.array([0., 1, 0, -2, 1, 0])
dirs = [np.array([1., 0, 0, 0, 0, 0]),
        np.array([0., 0, 1, 0, 0, 0]),
        np.array([0., -1, 0, -1, 1, 1])]
 
print("particular works:", np.allclose(A @ xp, b))
for k, d in enumerate(dirs, 1):
    print(f"  direction {k} in the kernel:", np.allclose(A @ d, 0))
combo = xp + 4 * dirs[0] - 2 * dirs[1] + 7 * dirs[2]
print("an arbitrary combination still solves:", np.allclose(A @ combo, b))

text

rk(A) = 3  rk(A|b) = 3  dim(S) = 3
columns 1 and 3 are entirely zero: True True
particular works: True
  direction 1 in the kernel: True
  direction 2 in the kernel: True
  direction 3 in the kernel: True
an arbitrary combination still solves: True

Exercise 2.7

Find all solutions $\mathbf{x} \in \mathbb{R}^3$ of $\mathbf{A}\mathbf{x} = 12\mathbf{x}$ where

$$\mathbf{A} = \begin{bmatrix}6&4&3\\6&0&9\\0&8&0\end{bmatrix}, \qquad \text{subject to} \quad \sum_{i=1}^{3}x_i = 1$$

Solution

Rearrange first. $\mathbf{A}\mathbf{x} = 12\mathbf{x}$ is the same as $(\mathbf{A} - 12\mathbf{I})\mathbf{x} = \mathbf{0}$ — a homogeneous system. (It is also an eigenvalue equation with $\lambda = 12$, which is §4.2’s subject; here it is just a linear system.)

$$\mathbf{A} - 12\mathbf{I} = \begin{bmatrix}-6&4&3\\6&-12&9\\0&8&-12\end{bmatrix}$$

This has rank 2, so its null space is one-dimensional — a line of solutions through the origin. The constraint $\sum_i x_i = 1$ then picks out a single point on that line, since the sum is a linear functional that is nonzero on the direction.

Stacking the constraint as a fourth row and solving gives

$$\boxed{\mathbf{x} = \begin{bmatrix}\tfrac38\\ \tfrac38\\ \tfrac14\end{bmatrix} = \begin{bmatrix}0.375\\ 0.375\\ 0.25\end{bmatrix}}$$

Verify by hand against the third row of the original equation, $8x_2 = 12x_3$: $8(\tfrac38) = 3$ and $12(\tfrac14) = 3$ ✓. And the sum: $\tfrac38 + \tfrac38 + \tfrac14 = 1$ ✓.

Note what the constraint did. Without it there are infinitely many solutions (the whole null-space line). The constraint is an inhomogeneous equation, so adding it turns a subspace into a single point — the intersection of a line through the origin with an affine plane (§2.8).

verify_2_7.py

import numpy as np
from fractions import Fraction
 
A = np.array([[6., 4, 3], [6, 0, 9], [0, 8, 0]])
 
# The rearranged homogeneous system, plus the constraint as an extra row.
M = np.r_[A - 12 * np.eye(3), np.ones((1, 3))]
rhs = np.r_[np.zeros(3), 1.0]
x = np.linalg.lstsq(M, rhs, rcond=None)[0]
 
print("rank of A - 12I:", np.linalg.matrix_rank(A - 12 * np.eye(3)),
      "-> null space is", 3 - np.linalg.matrix_rank(A - 12 * np.eye(3)), "dimensional")
print("solution:", x)
print("as fractions:", [str(Fraction(v).limit_denominator(200)) for v in x])
print("A@x == 12x :", np.allclose(A @ x, 12 * x))
print("sums to 1  :", np.isclose(x.sum(), 1.0))
print("12 is an eigenvalue of A:", np.any(np.isclose(np.linalg.eigvals(A).real, 12)))

text

rank of A - 12I: 2 -> null space is 1 dimensional
solution: [0.375 0.375 0.25 ]
as fractions: ['3/8', '3/8', '1/4']
A@x == 12x : True
sums to 1  : True
12 is an eigenvalue of A: True

Exercise 2.8

Determine the inverses of the following matrices if possible.

(a) $\begin{bmatrix}2&3&4\\3&4&5\\4&5&6\end{bmatrix}$   (b) $\begin{bmatrix}1&0&1&0\\0&1&1&0\\1&1&0&1\\1&1&1&0\end{bmatrix}$

Solution

(a) Not invertible. The rows are in arithmetic progression: row 2 minus row 1 is $(1,1,1)$, and so is row 3 minus row 2. Therefore

$$\text{row}_1 - 2\,\text{row}_2 + \text{row}_3 = \mathbf{0}$$

a non-trivial combination reaching zero, so the rows are linearly dependent (§2.5), $\operatorname{rk} = 2 < 3$, $\det = 0$, and by §2.6’s criterion there is no inverse.

Worth spotting the pattern rather than computing: any $3\times3$ matrix whose rows are three consecutive terms of an arithmetic progression is singular for exactly this reason.

(b) Invertible, with $\det = 1$. Augmenting with the identity and reducing (§2.3’s method) gives

$$\mathbf{A}^{-1} = \begin{bmatrix} 0 & -1 & 0 & 1\\ -1 & 0 & 0 & 1\\ 1 & 1 & 0 & -1\\ 1 & 1 & 1 & -2 \end{bmatrix}$$

Spot-check row 1 of $\mathbf{A}$ against column 1 of $\mathbf{A}^{-1}$: $(1,0,1,0)\cdot(0,-1,1,1) = 0 + 0 + 1 + 0 = 1$ ✓. And row 1 against column 2: $(1,0,1,0)\cdot(-1,0,1,1) = -1 + 0 + 1 + 0 = 0$ ✓.

verify_2_8.py

import numpy as np
 
Aa = np.array([[2., 3, 4], [3, 4, 5], [4, 5, 6]])
print("(a) det:", round(np.linalg.det(Aa), 12), " rank:", np.linalg.matrix_rank(Aa))
print("    row1 - 2*row2 + row3 =", Aa[0] - 2 * Aa[1] + Aa[2], "-> dependent")
try:
    np.linalg.inv(Aa)
except np.linalg.LinAlgError as e:
    print("    inv raises:", e)
 
Ab = np.array([[1., 0, 1, 0], [0, 1, 1, 0], [1, 1, 0, 1], [1, 1, 1, 0]])
inv = np.linalg.inv(Ab)
print("\n(b) det:", round(np.linalg.det(Ab), 12))
print("    inverse:\n", np.round(inv, 10))
print("    A @ inv == I:", np.allclose(Ab @ inv, np.eye(4)))
print("    entries are all integers:", np.allclose(inv, np.round(inv)))

text

(a) det: 0.0  rank: 2
    row1 - 2*row2 + row3 = [0. 0. 0.] -> dependent
    inv raises: Singular matrix
 
(b) det: 1.0
    inverse:
 [[ 0. -1.  0.  1.]
 [-1.  0.  0.  1.]
 [ 1.  1. -0. -1.]
 [ 1.  1.  1. -2.]]
    A @ inv == I: True
    entries are all integers: True

§2.4 — Subspaces

Exercise 2.9

Which of the following are subspaces of $\mathbb{R}^3$?

(a) $A = \{(\lambda, \lambda + \mu^3, \lambda - \mu^3) \mid \lambda, \mu \in \mathbb{R}\}$ (b) $B = \{(\lambda^2, -\lambda^2, 0) \mid \lambda \in \mathbb{R}\}$ (c) $C = \{(\xi_1, \xi_2, \xi_3) \in \mathbb{R}^3 \mid \xi_1 - 2\xi_2 + 3\xi_3 = \gamma\}$ for a fixed $\gamma \in \mathbb{R}$ (d) $D = \{(\xi_1, \xi_2, \xi_3) \in \mathbb{R}^3 \mid \xi_2 \in \mathbb{Z}\}$

Solution

(a) Yes, a subspace. The trick is that $\mu^3$ ranges over all of $\mathbb{R}$ as $\mu$ does — cubing is a bijection on $\mathbb{R}$, unlike squaring. So substituting $\nu = \mu^3$,

$$A = \{(\lambda, \lambda + \nu, \lambda - \nu) \mid \lambda, \nu \in \mathbb{R}\} = \operatorname{span}\left[\begin{bmatrix}1\\1\\1\end{bmatrix}, \begin{bmatrix}0\\1\\-1\end{bmatrix}\right]$$

A span is always a subspace (§2.6), and this one is two-dimensional.

(b) No. Here $\lambda^2 \ge 0$ always, so the first coordinate can never be negative. Take $\mathbf{v} = (1,-1,0) \in B$ (from $\lambda = 1$). Then $-\mathbf{v} = (-1,1,0)$ would need $\lambda^2 = -1$ — impossible. Not closed under scaling. It does contain $\mathbf{0}$ (at $\lambda = 0$), which is exactly why you must check all three conditions.

Contrast with (a): the difference is entirely that cubing is onto $\mathbb{R}$ and squaring is not.

(c) Only if $\gamma = 0$. For $\gamma = 0$ it is the null space of $[1, -2, 3]$, a two-dimensional subspace. For $\gamma \neq 0$ it does not contain $\mathbf{0}$ (since $0 - 0 + 0 = 0 \neq \gamma$), so it is an affine subspace — a plane not through the origin (§2.8), which is also a hyperplane in $\mathbb{R}^3$.

(d) No. Not closed under scaling: $(0,1,0) \in D$ but $\tfrac12(0,1,0) = (0, \tfrac12, 0)$ has $\xi_2 = \tfrac12 \notin \mathbb{Z}$. It is closed under addition (integers add to integers), and it does contain $\mathbf{0}$ — so it fails exactly one of the three conditions, and one is enough.

Summary of the failure modes, which is what this exercise is really teaching:

set	contains $\mathbf{0}$	closed under $+$	closed under scaling	subspace
$A$	yes	yes	yes	yes
$B$	yes	yes	no	no
$C$, $\gamma = 0$	yes	yes	yes	yes
$C$, $\gamma \neq 0$	no	no	no	no
$D$	yes	yes	no	no

verify_2_9.py

import numpy as np
 
rng = np.random.default_rng(0)
 
# (a) cubing is onto R, so A is a span -> a subspace. Check closure directly.
ok = True
for _ in range(5000):
    l1, m1, l2, m2 = rng.normal(size=4)
    v1 = np.array([l1, l1 + m1**3, l1 - m1**3])
    v2 = np.array([l2, l2 + m2**3, l2 - m2**3])
    s = v1 + v2
    # of the form (lam, lam+nu, lam-nu)?  lam = s0, nu = s1-s0, need s2 == s0-nu
    if not np.isclose(s[2], s[0] - (s[1] - s[0])):
        ok = False; break
print("(a) closed under addition:", ok)
print("    A == span{(1,1,1), (0,1,-1)}, rank:",
      np.linalg.matrix_rank(np.array([[1., 1, 1], [0, 1, -1]]).T))
 
# (b) the witness for the scaling failure.
v = np.array([1., -1, 0])          # lambda = 1
print("\n(b) v =", v, "is in B;  -v =", -v,
      "would need lambda^2 = -1 ->", "impossible")
 
# (c) depends on gamma.
w = np.array([1., -2, 3])
for gamma in (0.0, 5.0):
    print(f"\n(c) gamma={gamma}: origin in the set?", np.isclose(w @ np.zeros(3), gamma))
 
# (d) the witness for the scaling failure.
d = np.array([0., 1, 0])
print("\n(d) d =", d, "in D;  0.5*d =", 0.5 * d,
      "has second entry", 0.5, "-> not an integer")
print("    but d + d =", d + d, "IS in D, so addition is fine")

text

(a) closed under addition: True
    A == span{(1,1,1), (0,1,-1)}, rank: 2
 
(b) v = [ 1. -1.  0.] is in B;  -v = [-1.  1. -0.] would need lambda^2 = -1 -> impossible
 
(c) gamma=0.0: origin in the set? True
 
(c) gamma=5.0: origin in the set? False
 
(d) d = [0. 1. 0.] in D;  0.5*d = [0.  0.5 0. ] has second entry 0.5 -> not an integer
    but d + d = [0. 2. 0.] IS in D, so addition is fine

§2.5–2.6 — Independence, basis and rank

Exercise 2.10

Are the following sets of vectors linearly independent?

(a) $\mathbf{x}_1 = \begin{bmatrix}2\\-1\\3\end{bmatrix}$, $\mathbf{x}_2 = \begin{bmatrix}1\\1\\-2\end{bmatrix}$, $\mathbf{x}_3 = \begin{bmatrix}3\\-3\\8\end{bmatrix}$

(b) $\mathbf{x}_1 = \begin{bmatrix}1\\2\\1\\0\\0\end{bmatrix}$, $\mathbf{x}_2 = \begin{bmatrix}1\\1\\0\\1\\1\end{bmatrix}$, $\mathbf{x}_3 = \begin{bmatrix}1\\0\\0\\1\\1\end{bmatrix}$

Solution

(a) Linearly dependent. Writing the vectors as columns and eliminating gives rank 2, not 3. The relation is

$$\mathbf{x}_3 = 2\mathbf{x}_1 - \mathbf{x}_2$$

Check it entry by entry: $2(2) - 1 = 3$ ✓, $2(-1) - 1 = -3$ ✓, $2(3) - (-2) = 8$ ✓.

Equivalently $2\mathbf{x}_1 - \mathbf{x}_2 - \mathbf{x}_3 = \mathbf{0}$ — a non-trivial combination reaching zero, which is §2.5’s definition.

(b) Linearly independent, rank 3. Three vectors in $\mathbb{R}^5$, and the counting bound does not rule them out ($3 \le 5$), so the computation is necessary.

A fast hand argument: look at coordinates 3, 4 and 2 in turn. Only $\mathbf{x}_1$ has a nonzero third coordinate, so any combination reaching zero needs $\lambda_1 = 0$. Then the second coordinate gives $\lambda_2 \cdot 1 + \lambda_3 \cdot 0 = 0$, so $\lambda_2 = 0$. Then the first gives $\lambda_3 = 0$. Only the trivial solution — independent.

That kind of cascade argument is worth learning: it avoids elimination entirely when the sparsity pattern cooperates.

verify_2_10.py

import numpy as np
 
# (a)
x1 = np.array([2., -1, 3]); x2 = np.array([1., 1, -2]); x3 = np.array([3., -3, 8])
A = np.column_stack([x1, x2, x3])
print("(a) rank:", np.linalg.matrix_rank(A), "of 3 ->",
      "independent" if np.linalg.matrix_rank(A) == 3 else "DEPENDENT")
print("    2*x1 - x2 =", 2 * x1 - x2, " == x3:", np.allclose(2 * x1 - x2, x3))
print("    2*x1 - x2 - x3 =", 2 * x1 - x2 - x3, "-> non-trivial route to zero")
 
# (b)
y1 = np.array([1., 2, 1, 0, 0]); y2 = np.array([1., 1, 0, 1, 1]); y3 = np.array([1., 0, 0, 1, 1])
B = np.column_stack([y1, y2, y3])
print("\n(b) rank:", np.linalg.matrix_rank(B), "of 3 ->",
      "independent" if np.linalg.matrix_rank(B) == 3 else "DEPENDENT")
print("    only x1 has a nonzero third coordinate:", y1[2], y2[2], y3[2])

text

(a) rank: 2 of 3 -> DEPENDENT
    2*x1 - x2 = [ 3. -3.  8.]  == x3: True
    2*x1 - x2 - x3 = [0. 0. 0.] -> non-trivial route to zero
 
(b) rank: 3 of 3 -> independent
    only x1 has a nonzero third coordinate: 1.0 0.0 0.0

Exercise 2.11

Write $\mathbf{y} = \begin{bmatrix}1\\-2\\5\end{bmatrix}$ as a linear combination of

$$\mathbf{x}_1 = \begin{bmatrix}1\\1\\1\end{bmatrix},\quad \mathbf{x}_2 = \begin{bmatrix}1\\2\\3\end{bmatrix},\quad \mathbf{x}_3 = \begin{bmatrix}2\\-1\\1\end{bmatrix}$$

Solution

This is solving $\mathbf{X}\boldsymbol\lambda = \mathbf{y}$ with the $\mathbf{x}_i$ as columns. The matrix is invertible (rank 3), so the combination is unique — the $\mathbf{x}_i$ form a basis of $\mathbb{R}^3$ and §2.6’s characterisation 4 applies.

$$\begin{bmatrix}1&1&2\\1&2&-1\\1&3&1\end{bmatrix} \begin{bmatrix}\lambda_1\\\lambda_2\\\lambda_3\end{bmatrix} = \begin{bmatrix}1\\-2\\5\end{bmatrix}$$

Subtracting row 1 from rows 2 and 3:

$$\begin{aligned} \lambda_1 + \lambda_2 + 2\lambda_3 &= 1\\ \lambda_2 - 3\lambda_3 &= -3\\ 2\lambda_2 - \lambda_3 &= 4 \end{aligned}$$

From the last two: doubling the second gives $2\lambda_2 - 6\lambda_3 = -6$; subtracting from the third gives $5\lambda_3 = 10$, so $\lambda_3 = 2$. Then $\lambda_2 = -3 + 6 = 3$, and $\lambda_1 = 1 - 3 - 4 = -6$.

$$\boxed{\mathbf{y} = -6\mathbf{x}_1 + 3\mathbf{x}_2 + 2\mathbf{x}_3}$$

Check: $-6(1,1,1) + 3(1,2,3) + 2(2,-1,1) = (-6+3+4,\; -6+6-2,\; -6+9+2) = (1,-2,5)$ ✓

verify_2_11.py

import numpy as np
 
X = np.column_stack([[1., 1, 1], [1., 2, 3], [2., -1, 1]])
y = np.array([1., -2, 5])
 
print("rank of X:", np.linalg.matrix_rank(X), "-> a basis, so the combination is UNIQUE")
lam = np.linalg.solve(X, y)
print("coefficients:", lam)
print("rebuild:", X @ lam, " matches y:", np.allclose(X @ lam, y))
print("integers:", np.allclose(lam, np.round(lam)))

text

rank of X: 3 -> a basis, so the combination is UNIQUE
coefficients: [-6.  3.  2.]
rebuild: [ 1. -2.  5.]  matches y: True
integers: True

Exercise 2.12

Consider two subspaces of $\mathbb{R}^4$:

$$U_1 = \operatorname{span}\left[\begin{bmatrix}1\\2\\-3\\1\end{bmatrix}, \begin{bmatrix}2\\-1\\0\\-1\end{bmatrix}, \begin{bmatrix}-1\\1\\-1\\1\end{bmatrix}\right], \qquad U_2 = \operatorname{span}\left[\begin{bmatrix}-1\\-2\\2\\1\end{bmatrix}, \begin{bmatrix}2\\-2\\0\\0\end{bmatrix}, \begin{bmatrix}-3\\-2\\-2\\-1\end{bmatrix}\right]$$

Determine a basis of $U_1 \cap U_2$.

Solution

The method. A vector in the intersection is expressible both ways: $\mathbf{U}_1\boldsymbol\alpha = \mathbf{U}_2\boldsymbol\beta$ for some coefficient vectors. Rearranging,

$$\begin{bmatrix}\mathbf{U}_1 & -\mathbf{U}_2\end{bmatrix}\begin{bmatrix}\boldsymbol\alpha\\ \boldsymbol\beta\end{bmatrix} = \mathbf{0}$$

So compute the null space of the stacked matrix $[\mathbf{U}_1 \mid -\mathbf{U}_2]$ (a $4\times6$ matrix), and each null vector’s first three entries give an $\boldsymbol\alpha$ whose $\mathbf{U}_1\boldsymbol\alpha$ lies in the intersection.

Dimension first, as a sanity check. Both subspaces have rank 3, and $\operatorname{rk}[\mathbf{U}_1 \mid \mathbf{U}_2] = 4$, so by the dimension formula

$$\dim(U_1 \cap U_2) = \dim U_1 + \dim U_2 - \dim(U_1 + U_2) = 3 + 3 - 4 = 2$$

Two basis vectors expected. Carrying out the elimination and clearing denominators gives

$$U_1 \cap U_2 = \operatorname{span}\left[\begin{bmatrix}4\\-1\\-2\\-1\end{bmatrix}, \begin{bmatrix}-4\\-5\\2\\1\end{bmatrix}\right]$$

Any two independent vectors in the intersection would be an equally correct answer, so do not expect your basis to match this one — check instead that each of your vectors lies in both subspaces, and that you have two independent ones.

verify_2_12.py

import numpy as np
 
U1 = np.array([[1., 2, -1], [2, -1, 1], [-3, 0, -1], [1, -1, 1]])
U2 = np.array([[-1., 2, -3], [-2, -2, -2], [2, 0, -2], [1, 0, -1]])
 
d1, d2 = np.linalg.matrix_rank(U1), np.linalg.matrix_rank(U2)
dsum = np.linalg.matrix_rank(np.c_[U1, U2])
print("dim U1 =", d1, " dim U2 =", d2, " dim(U1+U2) =", dsum)
print("dim(U1 cap U2) = d1 + d2 - dsum =", d1 + d2 - dsum)
 
# The claimed basis.
basis = [np.array([4., -1, -2, -1]), np.array([-4., -5, 2, 1])]
 
def in_span(M, v, tol=1e-9):
    """Is v in the column space of M? Compare rank before and after appending."""
    return np.linalg.matrix_rank(M) == np.linalg.matrix_rank(np.c_[M, v])
 
for k, v in enumerate(basis, 1):
    print(f"  v{k} = {v}   in U1: {in_span(U1, v)}   in U2: {in_span(U2, v)}")
 
print("the two are independent:",
      np.linalg.matrix_rank(np.column_stack(basis)) == 2)
print("a combination is still in both:",
      in_span(U1, 3 * basis[0] - 2 * basis[1]),
      in_span(U2, 3 * basis[0] - 2 * basis[1]))

text

dim U1 = 3  dim U2 = 3  dim(U1+U2) = 4
dim(U1 cap U2) = d1 + d2 - dsum = 2
  v1 = [ 4. -1. -2. -1.]   in U1: True   in U2: True
  v2 = [-4. -5.  2.  1.]   in U1: True   in U2: True
the two are independent: True
a combination is still in both: True True

Exercise 2.15

Let $F = \{(x,y,z) \in \mathbb{R}^3 \mid x + y - z = 0\}$ and $G = \{(a-b,\, a+b,\, a-3b) \mid a, b \in \mathbb{R}\}$.

(a) Show that $F$ and $G$ are subspaces of $\mathbb{R}^3$. (b) Calculate $F \cap G$ without using any basis vector. (c) Find one basis for $F$ and one for $G$, calculate $F \cap G$ using them, and check against (b).

Solution

(a) $F$ is the null space of the matrix $[1, 1, -1]$, and a null space is always a subspace (§2.4). $G$ is a span — $G = \operatorname{span}[(1,1,1), (-1,1,-3)]$, reading off the coefficients of $a$ and $b$ — and a span is always a subspace. Both contain $\mathbf{0}$ (at $x=y=z=0$ and at $a=b=0$).

(b) Without any basis. A point of $G$ is $(a-b, a+b, a-3b)$. It lies in $F$ exactly when

$$(a-b) + (a+b) - (a-3b) = 0$$

Expanding: $a - b + a + b - a + 3b = a + 3b$. So the condition is $a + 3b = 0$, i.e. $a = -3b$. Substituting back:

$$(a-b,\, a+b,\, a-3b) = (-3b-b,\; -3b+b,\; -3b-3b) = (-4b,\, -2b,\, -6b) = -2b\,(2, 1, 3)$$$$\boxed{F \cap G = \operatorname{span}[(2,1,3)]}$$

a one-dimensional subspace. Note the whole calculation was one substitution — no elimination at all.

(c) With bases. For $F$, solve $x + y - z = 0$: two free variables, so $F = \operatorname{span}[(1,0,1), (0,1,1)]$, dimension 2. For $G$, the two spanning vectors $(1,1,1)$ and $(-1,1,-3)$ are independent, so they are a basis and $\dim G = 2$.

Now the dimension formula: $\dim(F+G) = 3$ (both are planes in $\mathbb{R}^3$ and they are distinct, so together they span everything), giving

$$\dim(F \cap G) = 2 + 2 - 3 = 1$$

matching (b). And $(2,1,3)$ satisfies $2 + 1 - 3 = 0$ ✓ so it is in $F$, and it equals $-\tfrac12\big((1,1,1) \cdot 1 + (-1,1,-3)\cdot(-3)\big)$… more simply, $a=-3, b=1$ gives $(-4,-2,-6) = -2(2,1,3)$ ✓ so it is in $G$.

verify_2_15.py

import numpy as np
 
# F is the null space of [1, 1, -1]; G is a span.
wF = np.array([1., 1, -1])
G = np.column_stack([[1., 1, 1], [-1., 1, -3]])
F = np.column_stack([[1., 0, 1], [0., 1, 1]])
 
print("dim F:", np.linalg.matrix_rank(F), " dim G:", np.linalg.matrix_rank(G))
print("dim(F+G):", np.linalg.matrix_rank(np.c_[F, G]))
print("dim(F cap G) =", np.linalg.matrix_rank(F) + np.linalg.matrix_rank(G)
      - np.linalg.matrix_rank(np.c_[F, G]))
 
v = np.array([2., 1, 3])
print("\nclaimed basis vector:", v)
print("  in F  (satisfies x+y-z=0):", np.isclose(wF @ v, 0))
print("  in G  (a=-3, b=1 gives -2v):", np.allclose(G @ np.array([-3., 1.]), -2 * v))
 
# Every G point that lies in F must have a + 3b = 0.
rng = np.random.default_rng(0)
checks = []
for _ in range(2000):
    a, b = rng.normal(size=2)
    pt = np.array([a - b, a + b, a - 3 * b])
    checks.append(np.isclose(wF @ pt, 0) == np.isclose(a + 3 * b, 0))
print("  'in F' is equivalent to a + 3b = 0:", all(checks))

text

dim F: 2  dim G: 2
dim(F+G): 3
dim(F cap G) = 1
 
claimed basis vector: [2. 1. 3.]
  in F  (satisfies x+y-z=0): True
  in G  (a=-3, b=1 gives -2v): True
  'in F' is equivalent to a + 3b = 0: True

§2.7 — Linear mappings

Exercise 2.16

Are the following mappings linear?

(a) $\Phi : L^1([a,b]) \to \mathbb{R}$, $f \mapsto \int_a^b f(x)\,\mathrm{d}x$ (b) $\Phi : C^1 \to C^0$, $f \mapsto f'$ (c) $\Phi : \mathbb{R} \to \mathbb{R}$, $x \mapsto \cos(x)$ (d) $\Phi : \mathbb{R}^3 \to \mathbb{R}^2$, $\mathbf{x} \mapsto \begin{bmatrix}1&2&3\\1&4&3\end{bmatrix}\mathbf{x}$ (e) $\Phi : \mathbb{R}^2 \to \mathbb{R}^2$, $\mathbf{x} \mapsto \begin{bmatrix}\cos\theta & \sin\theta\\ -\sin\theta & \cos\theta\end{bmatrix}\mathbf{x}$ for $\theta \in [0, 2\pi)$

Solution

(a) Linear. $\int_a^b (\lambda f + \psi g) = \lambda\int_a^b f + \psi\int_a^b g$ — integration is linear, and this is one of the reasons that fact matters. Note the domain is a space of functions, which are vectors by §2.4’s axioms.

(b) Linear. $(\lambda f + \psi g)' = \lambda f' + \psi g'$ — differentiation is linear too. Chapter 5 rests entirely on this.

(c) Not linear. $\cos(0) = 1 \neq 0$, and every linear mapping sends $\mathbf{0}$ to $\mathbf{0}$. One evaluation disqualifies it. (Additivity also fails: $\cos(x + y) \neq \cos x + \cos y$ in general.)

(d) Linear. Any mapping of the form $\mathbf{x} \mapsto \mathbf{A}\mathbf{x}$ is linear, because $\mathbf{A}(\lambda\mathbf{x} + \psi\mathbf{y}) = \lambda\mathbf{A}\mathbf{x} + \psi\mathbf{A}\mathbf{y}$ by distributivity (§2.2, Equation 2.19).

(e) Linear, for the same reason — it is $\mathbf{x} \mapsto \mathbf{R}\mathbf{x}$ with $\mathbf{R}$ a rotation matrix. The trigonometric functions appear in the entries, not applied to the variable, which is the whole distinction from (c). This is exactly the trap flagged on §2.1’s page: linearity is about the unknowns, not about how the coefficients were computed.

The pattern worth extracting: (a), (b), (d) and (e) are all linear, and (c) is not, and the difference is not “does a transcendental function appear” — $\cos$ appears in both (c) and (e). It is where it appears.

verify_2_16.py

import numpy as np
 
rng = np.random.default_rng(0)
 
def is_linear(phi, dim, trials=2000, tol=1e-9):
    for _ in range(trials):
        x, y = rng.normal(size=dim), rng.normal(size=dim)
        lam, psi = rng.normal(), rng.normal()
        if not np.allclose(phi(lam * x + psi * y),
                           lam * np.asarray(phi(x)) + psi * np.asarray(phi(y)), atol=tol):
            return False
    return True
 
# (a) integration, discretised as a Riemann sum on a fixed grid.
grid = np.linspace(0.0, 1.0, 400)
integrate = lambda f_vals: float(np.trapezoid(f_vals, grid))
print("(a) integral   linear:", is_linear(integrate, grid.size))
 
# (b) differentiation, as a finite difference on the same grid.
derivative = lambda f_vals: np.gradient(f_vals, grid)
print("(b) derivative linear:", is_linear(derivative, grid.size))
 
# (c) cos.
print("(c) cos        linear:", is_linear(np.cos, 3), "  cos(0) =", np.cos(0.0))
 
# (d) and (e) matrix multiplication.
Ad = np.array([[1., 2, 3], [1., 4, 3]])
theta = 1.1
Re = np.array([[np.cos(theta), np.sin(theta)], [-np.sin(theta), np.cos(theta)]])
print("(d) A @ x      linear:", is_linear(lambda v: Ad @ v, 3))
print("(e) R @ x      linear:", is_linear(lambda v: Re @ v, 2))
print("    (e) sends 0 to 0:", np.allclose(Re @ np.zeros(2), 0),
      " while (c) sends 0 to", np.cos(0.0))

text

(a) integral   linear: True
(b) derivative linear: True
(c) cos        linear: False   cos(0) = 1.0
(d) A @ x      linear: True
(e) R @ x      linear: True
    (e) sends 0 to 0: True  while (c) sends 0 to 1.0

Exercise 2.17

Consider the linear mapping

$$\Phi : \mathbb{R}^3 \to \mathbb{R}^4, \qquad \Phi\!\left(\begin{bmatrix}x_1\\x_2\\x_3\end{bmatrix}\right) = \begin{bmatrix}3x_1 + 2x_2 + x_3\\ x_1 + x_2 + x_3\\ x_1 - 3x_2\\ 2x_1 + 3x_2 + x_3\end{bmatrix}$$

Find the transformation matrix $\mathbf{A}_\Phi$, determine $\operatorname{rk}(\mathbf{A}_\Phi)$, and compute the kernel and image of $\Phi$ with their dimensions.

Solution

The transformation matrix is read straight off the coefficients — row $i$ of $\mathbf{A}_\Phi$ is the coefficient list of output $i$:

$$\mathbf{A}_\Phi = \begin{bmatrix}3&2&1\\1&1&1\\1&-3&0\\2&3&1\end{bmatrix}$$

Equivalently, column $j$ is $\Phi(\mathbf{e}_j)$ — check: $\Phi(\mathbf{e}_1) = (3,1,1,2)^\top$, the first column ✓.

Rank. Reducing to row-echelon form gives three pivots, so $\operatorname{rk}(\mathbf{A}_\Phi) = \mathbf{3}$. Since the matrix is $4\times3$, that is $\min(4,3) = 3$, so it has full rank.

Kernel. By rank-nullity (§2.7, Theorem 2.24),

$$\dim(\ker\Phi) = \dim(V) - \dim(\operatorname{Im}\Phi) = 3 - 3 = 0$$

so $\ker(\Phi) = \{\mathbf{0}\}$ — the trivial kernel, and therefore $\Phi$ is injective.

Image. $\dim(\operatorname{Im}\Phi) = \operatorname{rk}(\mathbf{A}_\Phi) = 3$, and $\operatorname{Im}(\Phi)$ is the span of the three columns — a three-dimensional subspace of $\mathbb{R}^4$. Since $3 < 4$, $\Phi$ is not surjective.

The summary: injective, not surjective, not bijective. This is the tall-matrix row of §2.7’s comparison table: nothing is lost, and not everything is reached.

verify_2_17.py

import numpy as np
 
A = np.array([[3., 2, 1], [1, 1, 1], [1, -3, 0], [2, 3, 1]])
 
# Column j must equal Phi(e_j).
phi = lambda x: np.array([3*x[0] + 2*x[1] + x[2],
                          x[0] + x[1] + x[2],
                          x[0] - 3*x[1],
                          2*x[0] + 3*x[1] + x[2]])
for j in range(3):
    e = np.eye(3)[j]
    print(f"Phi(e{j+1}) = {phi(e)}  == column {j}: {np.allclose(phi(e), A[:, j])}")
 
m, n = A.shape
r = np.linalg.matrix_rank(A)
print("\nrank:", r, " full rank:", r == min(m, n))
print("dim ker:", n - r, "-> kernel trivial:", (n - r) == 0)
print("dim Im :", r, " of codomain dimension", m, "-> surjective:", r == m)
print("injective:", (n - r) == 0, " surjective:", r == m, " bijective:", (n - r) == 0 and r == m)
print("rank-nullity:", (n - r), "+", r, "=", n)
 
# Nothing but zero maps to zero.
rng = np.random.default_rng(0)
worst = max(np.linalg.norm(A @ rng.normal(size=3)) for _ in range(5000))
print("smallest image norm over random nonzero inputs is comfortably positive:", worst > 0)

text

Phi(e1) = [3. 1. 1. 2.]  == column 0: True
Phi(e2) = [ 2.  1. -3.  3.]  == column 1: True
Phi(e3) = [1. 1. 0. 1.]  == column 2: True
 
rank: 3  full rank: True
dim ker: 0 -> kernel trivial: True
dim Im : 3  of codomain dimension 4 -> surjective: False
injective: True  surjective: False  bijective: False
rank-nullity: 0 + 3 = 3
smallest image norm over random nonzero inputs is comfortably positive: True

Exercise 2.19

Consider the endomorphism $\Phi : \mathbb{R}^3 \to \mathbb{R}^3$ whose transformation matrix with respect to the standard basis is

$$\mathbf{A}_\Phi = \begin{bmatrix}1&1&0\\1&-1&0\\1&1&1\end{bmatrix}$$

1. Determine $\ker(\Phi)$ and $\operatorname{Im}(\Phi)$. 2. Determine the transformation matrix $\tilde{\mathbf{A}}_\Phi$ with respect to the basis $B = \big((1,1,1)^\top, (1,2,1)^\top, (1,0,0)^\top\big)$.

Solution

1. $\det(\mathbf{A}_\Phi) = -2 \neq 0$, so the matrix is invertible and $\operatorname{rk}(\mathbf{A}_\Phi) = 3$. Therefore

$$\ker(\Phi) = \{\mathbf{0}\}, \qquad \operatorname{Im}(\Phi) = \mathbb{R}^3$$

Rank-nullity confirms: $0 + 3 = 3$ ✓. Since $\dim(V) = \dim(W) = 3$, §2.7’s three-way equivalence applies and $\Phi$ is injective, surjective and bijective — it is an automorphism.

Computing the determinant by expanding along the third column is quickest: the only nonzero entry there is the $1$ in position $(3,3)$, so $\det = 1 \cdot \det\begin{bmatrix}1&1\\1&-1\end{bmatrix} = 1(-1) - 1(1) = -2$.

2. Using §2.7’s Theorem 2.20 with $\mathbf{S} = \mathbf{T}$ (the same basis on both sides, since $\Phi$ is an endomorphism), so this is a similarity transformation (Definition 2.22):

$$\tilde{\mathbf{A}}_\Phi = \mathbf{S}^{-1}\mathbf{A}_\Phi\mathbf{S}, \qquad \mathbf{S} = \begin{bmatrix}1&1&1\\1&2&0\\1&1&0\end{bmatrix}$$

where the columns of $\mathbf{S}$ are the new basis vectors expressed in the old (standard) basis — which for the standard basis means just writing them down.

$$\tilde{\mathbf{A}}_\Phi = \begin{bmatrix}6&9&1\\-3&-5&0\\-1&-1&0\end{bmatrix}$$

Sanity check without redoing the arithmetic: similar matrices share their determinant and trace (§4.1 proves this; it follows from $\det(\mathbf{S}^{-1}\mathbf{A}\mathbf{S}) = \det\mathbf{A}$). Trace of $\mathbf{A}_\Phi$ is $1 - 1 + 1 = 1$; trace of $\tilde{\mathbf{A}}_\Phi$ is $6 - 5 + 0 = 1$ ✓. Determinant of both is $-2$ ✓. That pair of checks catches most arithmetic slips.

verify_2_19.py

import numpy as np
 
A = np.array([[1., 1, 0], [1, -1, 0], [1, 1, 1]])
 
print("det:", round(np.linalg.det(A), 12), " rank:", np.linalg.matrix_rank(A))
print("dim ker:", 3 - np.linalg.matrix_rank(A), " dim Im:", np.linalg.matrix_rank(A))
print("automorphism (injective, surjective, bijective):",
      np.linalg.matrix_rank(A) == 3)
 
# The new basis, as columns.
S = np.column_stack([[1., 1, 1], [1., 2, 1], [1., 0, 0]])
print("\nS =\n", S, "\ndet S:", round(np.linalg.det(S), 12), "-> a basis")
 
A_tilde = np.linalg.inv(S) @ A @ S
print("\nA_tilde = S^-1 A S =\n", np.round(A_tilde, 10))
 
# Similar matrices share determinant and trace.
print("\ntrace  A:", np.trace(A), "  A_tilde:", round(np.trace(A_tilde), 10))
print("det    A:", round(np.linalg.det(A), 10), "  A_tilde:", round(np.linalg.det(A_tilde), 10))
print("eigenvalues match:", np.allclose(np.sort(np.linalg.eigvals(A).real),
                                        np.sort(np.linalg.eigvals(A_tilde).real)))

text

det: -2.0  rank: 3
dim ker: 0  dim Im: 3
automorphism (injective, surjective, bijective): True
 
S =
 [[1. 1. 1.]
 [1. 2. 0.]
 [1. 1. 0.]] 
det S: 1.0 -> a basis
 
A_tilde = S^-1 A S =
 [[ 6.  9.  1.]
 [-3. -5.  0.]
 [-1. -1.  0.]]
 
trace  A: 1.0   A_tilde: 1.0
det    A: -2.0   A_tilde: -2.0
eigenvalues match: True

Exercise 2.20 (part 2)

With $\mathbf{b}_1 = \begin{bmatrix}2\\1\end{bmatrix}$, $\mathbf{b}_2 = \begin{bmatrix}-1\\-1\end{bmatrix}$, $\mathbf{b}'_1 = \begin{bmatrix}2\\-2\end{bmatrix}$, $\mathbf{b}'_2 = \begin{bmatrix}1\\1\end{bmatrix}$ and ordered bases $B = (\mathbf{b}_1, \mathbf{b}_2)$, $B' = (\mathbf{b}'_1, \mathbf{b}'_2)$ of $\mathbb{R}^2$:

1. Show that $B$ and $B'$ are both bases. 2. Compute the matrix $\mathbf{P}_1$ performing the basis change from $B'$ to $B$.

Solution

1. Both are bases iff the two vectors in each are independent, iff the determinant of the matrix with them as columns is nonzero.

$$\det\begin{bmatrix}2&-1\\1&-1\end{bmatrix} = -2 + 1 = -1 \neq 0, \qquad \det\begin{bmatrix}2&1\\-2&1\end{bmatrix} = 2 + 2 = 4 \neq 0$$

Both nonzero, so both are bases of $\mathbb{R}^2$.

2. $\mathbf{P}_1$ maps coordinates with respect to $B'$ onto coordinates with respect to $B$. Its $j$-th column is the coordinate vector of $\mathbf{b}'_j$ in the basis $B$ (§2.7, Definition 2.19). So we need $\mathbf{B}\mathbf{P}_1 = \mathbf{B}'$, giving

$$\mathbf{P}_1 = \mathbf{B}^{-1}\mathbf{B}' = \begin{bmatrix}2&-1\\1&-1\end{bmatrix}^{-1}\begin{bmatrix}2&1\\-2&1\end{bmatrix} = \begin{bmatrix}4&0\\6&-1\end{bmatrix}$$

Reading it off: the first column says $\mathbf{b}'_1 = 4\mathbf{b}_1 + 6\mathbf{b}_2$. Check: $4(2,1) + 6(-1,-1) = (8-6, 4-6) = (2,-2) = \mathbf{b}'_1$ ✓. The second says $\mathbf{b}'_2 = 0\mathbf{b}_1 - 1\mathbf{b}_2 = (1,1)$ ✓ — and indeed $\mathbf{b}'_2 = -\mathbf{b}_2$ exactly.

The direction to be careful about. $\mathbf{P}_1$ takes $B'$-coordinates to $B$-coordinates, and its columns are $B'$ vectors written in $B$. Those two statements sound like opposites and are the same thing — which is why this is the step people reverse. The check above ($\mathbf{B}\mathbf{P}_1 = \mathbf{B}'$) settles the direction unambiguously.

verify_2_20.py

import numpy as np
 
B  = np.column_stack([[2., 1], [-1., -1]])
Bp = np.column_stack([[2., -2], [1., 1]])
 
print("det B  =", round(np.linalg.det(B), 12), "-> a basis:", not np.isclose(np.linalg.det(B), 0))
print("det B' =", round(np.linalg.det(Bp), 12), "-> a basis:", not np.isclose(np.linalg.det(Bp), 0))
 
P1 = np.linalg.inv(B) @ Bp
print("\nP1 =\n", np.round(P1, 12))
print("B @ P1 == B' :", np.allclose(B @ P1, Bp))
 
# Column j of P1 expresses b'_j in the basis B.
for j in range(2):
    rebuilt = P1[0, j] * B[:, 0] + P1[1, j] * B[:, 1]
    print(f"  b'{j+1} = {P1[0, j]:g}*b1 + {P1[1, j]:g}*b2 = {rebuilt}  == {Bp[:, j]}")
 
# And it converts coordinates in the promised direction.
x = np.array([3.0, -2.0])                       # some vector
coords_Bp = np.linalg.solve(Bp, x)              # its coordinates in B'
coords_B  = np.linalg.solve(B, x)               # and in B
print("\ncoords in B' :", coords_Bp)
print("P1 @ coords in B' :", P1 @ coords_Bp)
print("coords in B       :", coords_B)
print("P1 maps B'-coords to B-coords:", np.allclose(P1 @ coords_Bp, coords_B))

text

det B  = -1.0 -> a basis: True
det B' = 4.0 -> a basis: True
 
P1 =
 [[ 4.  0.]
 [ 6. -1.]]
B @ P1 == B' : True
  b'1 = 4*b1 + 6*b2 = [ 2. -2.]  == [ 2. -2.]
  b'2 = 0*b1 + -1*b2 = [1. 1.]  == [1. 1.]
 
coords in B' : [1.25 0.5 ]
P1 @ coords in B' : [5. 7.]
coords in B       : [5. 7.]
P1 maps B'-coords to B-coords: True

Exercises not worked here

Five of the book’s twenty are left for you, and each is a variant of a technique above:

exercise	what it is	the technique it reuses
2.2	groups on congruence classes $\mathbb{Z}_n$, and when $\mathbb{Z}_n\setminus\{\bar0\}$ is a group under multiplication	Exercise 2.1’s axiom-by-axiom checking, plus Bézout’s theorem
2.13	$U_1, U_2$ as solution spaces of two homogeneous systems; find bases and $U_1 \cap U_2$	Exercise 2.12’s stacked-matrix method, after computing each null space
2.14	the same, with $U_1, U_2$ as column spans instead	Exercise 2.12 directly
2.18	abstract: $f \circ g = \mathrm{id}_E$ implies $\ker f = \ker(g\circ f)$, $\operatorname{Im} g = \operatorname{Im}(g\circ f)$, and $\ker f \cap \operatorname{Im} g = \{\mathbf{0}\}$	§2.7’s kernel and image definitions; no computation, pure reasoning
2.20 (parts 3–6)	a homomorphism $\mathbb{R}^2 \to \mathbb{R}^3$ specified on $\mathbf{b}_1 \pm \mathbf{b}_2$, then a full basis change	Exercises 2.19 and 2.20 part 2 combined

2.18 is the most valuable of the five because it is the only one with no arithmetic. A hint: for $\ker f = \ker(g \circ f)$, one inclusion is immediate and the other needs you to apply $f$ to both sides and use $f \circ g = \mathrm{id}$ — note the order of composition carefully, as it is $f \circ g$ that is the identity, not $g \circ f$.

pch.quizTag Did the exercises land?

1. Exercise 2.5(a) has no solution. What told you, before finding any x?

   A The matrix was not square B The rank of A was 3 while the rank of the augmented matrix was 4 C The determinant was zero D There were more unknowns than equations
   pch.quizShowAnswer

   B — The rank of A was 3 while the rank of the augmented matrix was 4 — Appending b raised the rank, so b lies outside the reach of the columns. That is section 2.6's solvability criterion, and it decides the question without solving anything.

2. In exercise 2.9, set A involves mu cubed and set B involves lambda squared. Only A is a subspace. Why?

   A Because cubes are larger than squares B Because cubing is onto all of R while squaring only reaches the non-negatives, so B is not closed under scaling by a negative C Because A has two parameters and B has one D Because B does not contain the zero vector
   pch.quizShowAnswer

   B — Because cubing is onto all of R while squaring only reaches the non-negatives, so B is not closed under scaling by a negative — B does contain zero, which is why all three conditions must be checked. The substitution nu equals mu cubed is legitimate precisely because cubing is a bijection on R; the same substitution for squaring is not.

3. Exercise 2.6's matrix has two entirely zero columns. What is the consequence?

   A The system is inconsistent B Those two variables are free, because they appear in no equation — so the solution set is three-dimensional rather than one-dimensional C Those columns can be deleted without changing the answer D The rank is zero
   pch.quizShowAnswer

   B — Those two variables are free, because they appear in no equation — so the solution set is three-dimensional rather than one-dimensional — They are genuine variables that are simply unconstrained. Reducing only the interesting block and reporting a unique solution is the classic error on this exercise.

4. In exercise 2.19, how can you check the basis-change result without redoing the multiplication?

   A Confirm the new matrix is diagonal B Compare the trace and determinant, which similar matrices share C Check the new matrix is symmetric D There is no shortcut
   pch.quizShowAnswer

   B — Compare the trace and determinant, which similar matrices share — Similar matrices have the same determinant, trace and eigenvalues. Trace one and determinant minus two match on both sides, which catches most arithmetic slips in seconds.

Recall card

• Exercise 2.1’s whole difficulty is spotting that the operation is (a+1)(b+1) minus 1 — multiplication in disguise, which is why minus one is the excluded element and why the neutral element is zero rather than one.
• The Heisenberg group of exercise 2.3 fails commutativity only in its top-right entry, because the product’s third parameter picks up an asymmetric cross term.
• Exercise 2.4 part (a) is undefined and parts (d) and (e) have different shapes — the two lessons of §2.2 in one exercise.
• Inconsistency is detected by comparing the rank of A with the rank of the augmented matrix, which is how 2.5(a) is settled without solving.
• Zero columns give free variables. Exercise 2.6 has two, and missing them turns a three-dimensional solution set into a point.
• An inhomogeneous constraint turns a null-space line into a single point — exercise 2.7’s requirement that the entries sum to one intersects a subspace with an affine plane.
• Rows in arithmetic progression are always dependent, which settles exercise 2.8(a) by inspection.
• Cubing is onto the reals and squaring is not, which is the entire difference between the two sets in exercise 2.9.
• Sparsity can replace elimination: exercise 2.10(b) is settled by a cascade argument on the coordinates only one vector touches.
• The intersection of two subspaces is the null space of the two spanning matrices stacked side by side, the second negated, and the dimension formula predicts its size in advance.
• Linearity is about where the transcendental function appears: cosine applied to the variable breaks it, cosine in the matrix entries does not.
• Similar matrices share trace, determinant and eigenvalues, which is the fastest check on any basis-change computation.
• A basis-change matrix’s columns are the new basis vectors written in the old basis, and checking that the old basis matrix times P gives the new one settles the direction whenever you doubt it.

Next: everything in one place — Chapter 2 Formula Sheet.