An orthogonal projection finds the point in a subspace that is closest to your vector — its “shadow.” This one operation is the workhorse of applied linear algebra: PCA projects data onto a low-dimensional subspace, least-squares regression projects targets onto the column space of the features, and lossy compression projects signals onto a small basis. Master projections and you understand the geometry of half of machine learning.

A real-life example: the shadow on the floor

Hold a pencil above the floor and shine a light straight down. The shadow is the pencil’s orthogonal projection onto the floor (a 2-D subspace). Two things are always true: the shadow lies in the floor, and the line from the pencil tip to its shadow is perpendicular to the floor. Those two properties define the projection — and they’re what make it the closest point.

Projection onto a line (1-D)

To project $\mathbf{x}$ onto the line $U$ spanned by a vector $\mathbf{b}$, we seek the multiple of $\mathbf{b}$ closest to $\mathbf{x}$. “Closest” means the error $\mathbf{x} - \pi_U(\mathbf{x})$ is orthogonal to $\mathbf{b}$. Solving that condition gives:

The last object $P_\pi$ is the projection matrix: multiply any vector by it and you get its projection onto the line. Projection matrices are symmetric and idempotent: $P_\pi^2 = P_\pi$ (projecting twice does nothing new — the shadow of a shadow is itself).

See the shadow

The white vector $\mathbf{x}$ orbits. Its projection $\pi_U(\mathbf{x})$ (green) slides along the line $U$, always the closest point; the red segment is the error $\mathbf{x} - \pi_U(\mathbf{x})$, always perpendicular to $U$:

The error is always perpendicular — that’s the defining property, and it’s why the projection is the least-error approximation of $\mathbf{x}$ inside $U$.

Projection onto a general subspace

For a subspace $U$ with basis vectors as columns of $B$, the same “error is orthogonal to the subspace” condition gives the normal equation $B^\top B\,\boldsymbol\lambda = B^\top\mathbf{x}$, hence:

The matrix $(B^\top B)^{-1}B^\top$ is the pseudo-inverse of $B$ — the exact object behind least-squares regression. If the basis is orthonormal ($B^\top B = I$), this collapses to the cheap form $\pi_U(\mathbf{x}) = B B^\top\mathbf{x}$ — no inverse needed (the ONB payoff again).

  flowchart TD
  X["vector x"] --> C["error x − π must be ⟂ to U"]
  C --> NE["normal equation
BᵀB λ = Bᵀx"]
  NE --> P["π_U(x) = B(BᵀB)⁻¹Bᵀ x"]
  P --> LS["least-squares regression (Ch 9)"]
  P --> PCA["PCA reconstruction (Ch 10)"]
  P --> ONB["ONB: simplifies to BBᵀx"]

NumPy

import numpy as np
 
# ---- 1-D: project onto a line spanned by b ----
b = np.array([1.0, 2.0, 2.0])
x = np.array([1.0, 1.0, 1.0])
P_line = np.outer(b, b) / (b @ b)          # projection matrix bbᵀ / bᵀb
proj = P_line @ x
print("projection onto line:", np.round(proj, 4))
print("idempotent P² = P:", np.allclose(P_line @ P_line, P_line))
print("error ⟂ b:", np.isclose((x - proj) @ b, 0))
 
# ---- general subspace: project onto column space of B ----
B = np.array([[1.0, 0.0],
              [1.0, 1.0],
              [1.0, 2.0]])
P = B @ np.linalg.inv(B.T @ B) @ B.T        # B(BᵀB)⁻¹Bᵀ
y = np.array([6.0, 0.0, 0.0])
print("projection onto plane:", np.round(P @ y, 4))

import numpy as np
 
# ---- 1-D: project onto a line spanned by b ----
b = np.array([1.0, 2.0, 2.0])
x = np.array([1.0, 1.0, 1.0])
P_line = np.outer(b, b) / (b @ b)          # projection matrix bbᵀ / bᵀb
proj = P_line @ x
print("projection onto line:", np.round(proj, 4))
print("idempotent P² = P:", np.allclose(P_line @ P_line, P_line))
print("error ⟂ b:", np.isclose((x - proj) @ b, 0))
 
# ---- general subspace: project onto column space of B ----
B = np.array([[1.0, 0.0],
              [1.0, 1.0],
              [1.0, 2.0]])
P = B @ np.linalg.inv(B.T @ B) @ B.T        # B(BᵀB)⁻¹Bᵀ
y = np.array([6.0, 0.0, 0.0])
print("projection onto plane:", np.round(P @ y, 4))

projection onto line: [0.5556 1.1111 1.1111]
idempotent P² = P: True
error ⟂ b: True
projection onto plane: [ 5.  2. -1.]

projection onto line: [0.5556 1.1111 1.1111]
idempotent P² = P: True
error ⟂ b: True
projection onto plane: [ 5.  2. -1.]

Why this matters for ML

• Linear regression is a projection: it projects the target vector onto the column space of the design matrix; the residual is the perpendicular error.
• PCA projects data onto the subspace spanned by the top principal components; the projection error is the reconstruction loss you minimize.
• Lossy compression keeps a few orthogonal-basis coefficients — a projection onto a small subspace.

🧪 Try It Yourself

Exercise 1 – Project onto a line

Exercise 2 – The error is perpendicular

Exercise 3 – Projection is idempotent

Recap

• An orthogonal projection is the closest point in a subspace; the error is perpendicular to the subspace.
• 1-D: $P_\pi = \mathbf{b}\mathbf{b}^\top/\lVert\mathbf{b}\rVert^2$. General: $P_\pi = B(B^\top B)^{-1}B^\top$ (the pseudo-inverse); ONB simplifies to $BB^\top$.
• Projection matrices are symmetric and idempotent ($P^2 = P$).
• Projections power regression, PCA, and compression.

Next: transformations that move data without distorting it — rotations.