The SVD writes a matrix as a sum of rank-1 pieces, ordered by importance. Keep only the top few and you get the best possible low-rank approximation — a rigorous, optimal form of compression. This is how you shrink an image, denoise data, and understand what PCA is really doing: throwing away the small singular values.

A real-life example: compressing a photo

A grayscale photo is a big matrix of pixel values. Its SVD reveals that a handful of singular values carry most of the visual structure and the rest are near-noise. Storing just the top $k$ singular values and vectors reconstructs a recognizable image using a tiny fraction of the storage. A rank-5 approximation of a 1432×1910 image needs ~0.6% of the numbers — and you can still see the scene.

A matrix as a sum of rank-1 pieces

From the SVD $A = U\Sigma V^\top$, expand the product:

Each $A_i$ is a rank-1 matrix (an outer product), weighted by its singular value $\sigma_i$. Since $\sigma_1 \ge \sigma_2 \ge \dots$, the first terms matter most. Truncating at $k$ gives the rank-$k$ approximation:

Watch the image rebuild

The heatmap below is reconstructed one rank-1 layer at a time. At rank 1 it’s a blurry blob; each extra singular value adds detail. The bars show the singular values — notice how the first few dominate, which is why a low-rank approximation works so well:

The Eckart-Young theorem: it’s optimal

Truncating the SVD isn’t just a low-rank approximation — it’s the best one. Measuring error with the spectral norm $\lVert A \rVert_2 = \sigma_1$ (the largest singular value = the most a matrix can stretch a vector), the Eckart-Young theorem states:

No rank-$k$ matrix approximates $A$ better than the truncated SVD, and the leftover error is exactly the first singular value you dropped, $\sigma_{k+1}$. That’s a remarkably clean, provable optimality guarantee.

  flowchart LR
  A["A = Σ σᵢ uᵢvᵢᵀ
(rank r)"] --> T["keep top k terms"]
  T --> AK["Â(k), rank k"]
  AK --> E["error = σ(k+1)  (Eckart-Young: optimal)"]
  AK -.-> USE["compression · denoising · PCA · latent factors"]

NumPy

import numpy as np
 
# A rank-3 matrix (built to have exactly 3 nonzero singular values)
rng = np.random.default_rng(0)
A = rng.standard_normal((8, 3)) @ rng.standard_normal((3, 6))
 
U, s, Vt = np.linalg.svd(A, full_matrices=False)
print("singular values:", np.round(s, 3))
 
def rank_k(k):
    return (U[:, :k] * s[:k]) @ Vt[:k, :]        # Σ_{i<k} σ_i u_i v_iᵀ
 
# Eckart-Young: error of best rank-k approx equals σ_{k+1}
for k in [1, 2, 3]:
    err = np.linalg.norm(A - rank_k(k), 2)       # spectral norm
    nxt = s[k] if k < len(s) else 0.0
    print(f"rank {k}: spectral error {err:.4f}  vs  σ_{k+1} = {nxt:.4f}")

import numpy as np
 
# A rank-3 matrix (built to have exactly 3 nonzero singular values)
rng = np.random.default_rng(0)
A = rng.standard_normal((8, 3)) @ rng.standard_normal((3, 6))
 
U, s, Vt = np.linalg.svd(A, full_matrices=False)
print("singular values:", np.round(s, 3))
 
def rank_k(k):
    return (U[:, :k] * s[:k]) @ Vt[:k, :]        # Σ_{i<k} σ_i u_i v_iᵀ
 
# Eckart-Young: error of best rank-k approx equals σ_{k+1}
for k in [1, 2, 3]:
    err = np.linalg.norm(A - rank_k(k), 2)       # spectral norm
    nxt = s[k] if k < len(s) else 0.0
    print(f"rank {k}: spectral error {err:.4f}  vs  σ_{k+1} = {nxt:.4f}")

singular values: [7.043 3.83  1.982 0.    0.    0.   ]
rank 1: spectral error 3.8300  vs  σ_2 = 3.8300
rank 2: spectral error 1.9820  vs  σ_3 = 1.9820
rank 3: spectral error 0.0000  vs  σ_4 = 0.0000

singular values: [7.043 3.83  1.982 0.    0.    0.   ]
rank 1: spectral error 3.8300  vs  σ_2 = 3.8300
rank 2: spectral error 1.9820  vs  σ_3 = 1.9820
rank 3: spectral error 0.0000  vs  σ_4 = 0.0000

The error at each rank is exactly the next singular value — Eckart-Young, confirmed.

Why this matters for ML

• Compression & denoising: dropping small singular values removes noise (which lives in the tail) while keeping signal — used on images, sensor data, embeddings.
• PCA is a low-rank approximation of the (centered) data / covariance matrix — keep the top components.
• Latent factor models (recommenders, LSA) are low-rank approximations of huge sparse matrices.
• Model compression: factorizing weight matrices into low rank shrinks and speeds up networks.

🧪 Try It Yourself

Exercise 1 – Build a rank-k approximation

Exercise 2 – Eckart-Young error

Exercise 3 – Captured energy

Recap

• The SVD writes $A$ as a sum of rank-1 pieces $\sigma_i\mathbf{u}_i\mathbf{v}_i^\top$, ordered by singular value.
• The rank-$k$ approximation keeps the top $k$; by Eckart-Young it’s the optimal rank-$k$ approximation in spectral norm, with error $\sigma_{k+1}$.
• This is the math of compression, denoising, PCA, and latent-factor models.

Next: a family tree tying every matrix type together — Matrix Phylogeny.