Matrix Decompositions — Overview
Chapters 2 and 3 gave us matrices and the geometry around them. Chapter 4 asks the deepest question yet: can we break a matrix into simpler, more revealing pieces? Just as exposes a number’s structure, a matrix decomposition factors a matrix into parts that expose its structure — the directions it stretches, the amount it scales, whether it can be inverted.
These factorizations are not academic. The SVD compresses images and powers recommender systems; eigendecomposition is the heart of PCA and Google’s PageRank; Cholesky makes Gaussian sampling and covariance math fast. This chapter is where linear algebra becomes the engine room of machine learning.
The concept map
flowchart TD DET["Determinant
& Trace"] INV["Invertibility"] CHOL["Cholesky"] EIG["Eigenvalues &
Eigenvectors"] DIAG["Eigendecomposition
A = PDP⁻¹"] SVD["SVD
A = UΣVᵀ"] APX["Low-rank
approximation"] DET -->|"tests"| INV INV -->|"used in"| CHOL DET -->|"used in"| EIG EIG -->|"constructs"| DIAG EIG --> SVD DIAG --> SVD SVD --> APX DIAG -.-> PCA["PCA (Ch 10)"] SVD -.-> PCA CHOL -.-> GAUSS["Gaussian models (Ch 6, 11)"] APX -.-> COMP["compression, recommenders"]
(Adapted from Figure 4.1 of the book.)
The through-line
- Determinant and Trace — two single numbers that summarize a matrix: signed volume and the sum of the diagonal. They test invertibility and stay fixed under basis change.
- Eigenvalues and Eigenvectors — the special directions a matrix only stretches, and the stretch factors. The “eigen-analysis” of a transformation.
- Cholesky Decomposition — a “square root” for symmetric positive-definite matrices, , used for covariance and sampling.
- Eigendecomposition and Diagonalization — : change to the eigenbasis, scale, change back. The clean form of a matrix.
- Singular Value Decomposition — , the universal decomposition that works for any matrix, square or not.
- Matrix Approximation — keep the top singular values to get the best low-rank approximation (Eckart-Young); this is compression and PCA.
- Matrix Phylogeny — a family tree tying every matrix type together: square, invertible, normal, symmetric, orthogonal, diagonal.
Where each idea shows up in ML
| Decomposition idea | Machine learning payoff |
|---|---|
| Determinant | Invertibility, Jacobians in normalizing flows, Gaussian densities |
| Eigenvalues/vectors | PCA directions, PageRank, spectral clustering, stability |
| Cholesky | Sampling from Gaussians, fast covariance solves, the reparameterization trick |
| Eigendecomposition | PCA, spectral methods, matrix powers |
| SVD | Recommender systems, LSA/topic models, pseudo-inverse, whitening |
| Low-rank approximation | Image/model compression, denoising, latent factors |
Prerequisites
- Chapter 2 — Linear Algebra: matrices, rank, basis, linear maps, null space.
- Chapter 3 — Analytic Geometry: orthonormal bases, orthogonal matrices, projections.
Start with Determinant and Trace.
If this helped you, consider buying me a coffee ☕
Buy me a coffeeWas this page helpful?
Let us know how we did