The Singular Value Decomposition (SVD) is often called the fundamental theorem of linear algebra because it works for every matrix — square or rectangular, invertible or not. It factors any matrix into a rotation, a scaling, and another rotation: $A = U\Sigma V^\top$. This one decomposition powers recommender systems, topic modeling, the pseudo-inverse, image compression, and PCA.

A real-life example: Netflix-style recommendations

Take a big users × movies ratings table (mostly empty). Its SVD uncovers hidden “taste factors” — a right-singular vector might represent “sci-fi lover,” a left-singular vector “stereotypical sci-fi movie,” and the singular value how strong that factor is. Reconstructing the table from the top few factors fills in the blanks — predicting ratings for movies a user hasn’t seen. That’s collaborative filtering, and it’s SVD.

The decomposition

For any $A \in \mathbb{R}^{m\times n}$ of rank $r$:

where

• $U \in \mathbb{R}^{m\times m}$ is orthogonal; its columns $\mathbf{u}_i$ are the left-singular vectors.
• $V \in \mathbb{R}^{n\times n}$ is orthogonal; its columns $\mathbf{v}_j$ are the right-singular vectors.
• $\Sigma \in \mathbb{R}^{m\times n}$ is “diagonal” with the singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$ on the diagonal and zeros elsewhere.

Because $U$ and $V$ are orthogonal (rotations) and $\Sigma$ is a pure scaling, the SVD says every linear map is a rotation, then an axis-aligned stretch, then another rotation.

The three stages, visualized

Apply $A = U\Sigma V^\top$ to the unit circle in three steps. $V^\top$ rotates it (watch the colored markers turn), $\Sigma$ scales it into an axis-aligned ellipse, and $U$ rotates the ellipse into its final orientation:

How it’s built

The SVD is constructed from two eigendecompositions:

• The right-singular vectors $V$ are the eigenvectors of the symmetric matrix $A^\top A$.
• The singular values are $\sigma_i = \sqrt{\lambda_i}$, the square roots of the eigenvalues of $A^\top A$ (which are $\ge 0$).
• The left-singular vectors are $\mathbf{u}_i = \frac{1}{\sigma_i}A\mathbf{v}_i$ (equivalently, eigenvectors of $AA^\top$).

This gives the singular value equation $A\mathbf{v}_i = \sigma_i\mathbf{u}_i$ — like the eigenvalue equation, but with different orthonormal bases on each side.

SVD vs. eigendecomposition

  flowchart TD
  E["Eigendecomposition
A = PDP⁻¹"] --> E1["square matrices only"]
  E --> E2["needs a basis of eigenvectors"]
  E --> E3["P generally NOT orthogonal"]
  S["SVD
A = UΣVᵀ"] --> S1["ANY m×n matrix"]
  S --> S2["always exists"]
  S --> S3["U, V orthogonal; Σ ≥ 0 real"]
  S3 -.-> SAME["for symmetric A, SVD = eigendecomposition"]

The key differences: the SVD always exists and applies to non-square matrices, and its factors are genuine rotations. For a symmetric positive-definite matrix, the two decompositions coincide.

NumPy

import numpy as np
 
A = np.array([[1.0, 0.0, 1.0],
              [-2.0, 1.0, 0.0]])          # 2x3, not square
 
U, s, Vt = np.linalg.svd(A)               # s is the vector of singular values
print("singular values:", np.round(s, 4))
print("U orthogonal:", np.allclose(U @ U.T, np.eye(2)))
print("V orthogonal:", np.allclose(Vt @ Vt.T, np.eye(3)))
 
# rebuild A = U Σ Vᵀ  (Σ padded to 2x3)
Sigma = np.zeros((2, 3))
Sigma[:2, :2] = np.diag(s)
print("A = U Σ Vᵀ:", np.allclose(U @ Sigma @ Vt, A))
 
# right-singular vectors are eigenvectors of AᵀA; σ² are its eigenvalues
w = np.linalg.eigvalsh(A.T @ A)
print("σ² ≈ eigenvalues of AᵀA:", np.round(np.sort(w)[::-1][:2], 4), "vs", np.round(s**2, 4))

import numpy as np
 
A = np.array([[1.0, 0.0, 1.0],
              [-2.0, 1.0, 0.0]])          # 2x3, not square
 
U, s, Vt = np.linalg.svd(A)               # s is the vector of singular values
print("singular values:", np.round(s, 4))
print("U orthogonal:", np.allclose(U @ U.T, np.eye(2)))
print("V orthogonal:", np.allclose(Vt @ Vt.T, np.eye(3)))
 
# rebuild A = U Σ Vᵀ  (Σ padded to 2x3)
Sigma = np.zeros((2, 3))
Sigma[:2, :2] = np.diag(s)
print("A = U Σ Vᵀ:", np.allclose(U @ Sigma @ Vt, A))
 
# right-singular vectors are eigenvectors of AᵀA; σ² are its eigenvalues
w = np.linalg.eigvalsh(A.T @ A)
print("σ² ≈ eigenvalues of AᵀA:", np.round(np.sort(w)[::-1][:2], 4), "vs", np.round(s**2, 4))

singular values: [2.4495 1.4142]
U orthogonal: True
V orthogonal: True
A = U Σ Vᵀ: True
σ² ≈ eigenvalues of AᵀA: [6. 2.] vs [6. 2.]

singular values: [2.4495 1.4142]
U orthogonal: True
V orthogonal: True
A = U Σ Vᵀ: True
σ² ≈ eigenvalues of AᵀA: [6. 2.] vs [6. 2.]

Why this matters for ML

• Recommender systems (collaborative filtering) factor the ratings matrix via SVD to find latent taste factors and predict missing entries.
• Latent Semantic Analysis / topic models apply SVD to term-document matrices to surface topics.
• Pseudo-inverse & least squares, PCA, and whitening are all computed most stably through the SVD.

🧪 Try It Yourself

Exercise 1 – Decompose any matrix

Exercise 2 – Reconstruct A = U Σ Vᵀ

Exercise 3 – Singular values vs eigenvalues of AᵀA

Recap

• The SVD $A = U\Sigma V^\top$ factors any matrix into rotation × scaling × rotation, with orthogonal $U, V$ and non-negative singular values in $\Sigma$.
• It always exists (unlike eigendecomposition) and handles non-square matrices.
• Built from the eigendecompositions of $A^\top A$ and $AA^\top$; $\sigma_i = \sqrt{\lambda_i}$.
• Powers recommenders, LSA, the pseudo-inverse, PCA, and whitening.

Next: keep only the top singular values for the best low-rank fit — Matrix Approximation.