Most vectors, when hit by a matrix, get both rotated and stretched. But a few special directions get only stretched — the matrix leaves their line alone and just scales them. Those directions are eigenvectors, and the scale factors are eigenvalues. This “eigen-analysis” reveals the true axes of a transformation and is the mathematical core of PCA, spectral clustering, PageRank, and the stability analysis of dynamical systems.

A real-life example: PageRank

Google’s original PageRank models web surfing as a giant matrix $A$ of click probabilities. Multiply an importance vector by $A$ over and over and it converges to a special vector that $A$ merely scales — the eigenvector with the largest eigenvalue. That steady-state vector is the ranking of every web page. Eigenvectors are, literally, what pages Google shows you first.

The eigenvalue equation

For a square matrix $A$, a nonzero vector $\mathbf{x}$ is an eigenvector with eigenvalue $\lambda$ if

In words: applying $A$ to $\mathbf{x}$ gives back the same direction, just scaled by $\lambda$. If $\lambda > 1$ it stretches, $0 < \lambda < 1$ it shrinks, $\lambda < 0$ it flips.

Eigenvectors are the directions that don’t turn

Watch a probe vector rotate around the circle. For a generic direction the output $A\mathbf{x}$ (green) points somewhere different from the input (white) — the matrix rotated it. But along the eigenvector lines (amber), the output stays perfectly parallel to the input, only longer or shorter. Those aligned moments are the eigenvectors:

Finding them: the characteristic polynomial

$\lambda$ is an eigenvalue exactly when $A - \lambda I$ is singular, i.e. when

The roots of this characteristic polynomial are the eigenvalues. For each $\lambda$, the eigenvectors are the nonzero solutions of $(A - \lambda I)\mathbf{x} = \mathbf{0}$ — i.e. the null space of $A - \lambda I$, called the eigenspace $E_\lambda$.

Multiplicities and defective matrices

• The algebraic multiplicity of $\lambda$ is how many times it’s a root of $p_A$.
• The geometric multiplicity is $\dim E_\lambda$ — how many independent eigenvectors it has.
• Geometric $\le$ algebraic, always. When a matrix has fewer than $n$ independent eigenvectors it’s called defective and cannot be diagonalized.

  flowchart TD
  A["Square matrix A"] --> CP["det(A − λI) = 0"]
  CP --> L["eigenvalues λᵢ (roots)"]
  L --> ES["eigenspace Eλ = null(A − λI)"]
  ES --> N["n independent eigenvectors?"]
  N -->|yes| DIAG["diagonalizable"]
  N -->|no| DEF["defective (not diagonalizable)"]

The spectral theorem

A special, beautiful case: if $A$ is symmetric ($A = A^\top$), then it always has real eigenvalues and an orthonormal basis of eigenvectors. Symmetric matrices are never defective. This is why covariance matrices (symmetric, positive semi-definite) always decompose cleanly — the guarantee that makes PCA work.

NumPy

import numpy as np
 
A = np.array([[2.0, 1.0],
              [1.0, 2.0]])
 
vals, vecs = np.linalg.eig(A)
print("eigenvalues:", np.round(vals, 4))          # [1. 3.]  (order may vary)
print("eigenvectors (columns):\n", np.round(vecs, 4))
 
# verify A x = λ x for the first eigenpair
lam, x = vals[0], vecs[:, 0]
print("A x =", np.round(A @ x, 4))
print("λ x =", np.round(lam * x, 4))
print("match:", np.allclose(A @ x, lam * x))
 
# symmetric matrices: use eigh for real eigenvalues + orthonormal eigenvectors
w, Q = np.linalg.eigh(A)
print("orthonormal eigenvectors:", np.allclose(Q.T @ Q, np.eye(2)))

import numpy as np
 
A = np.array([[2.0, 1.0],
              [1.0, 2.0]])
 
vals, vecs = np.linalg.eig(A)
print("eigenvalues:", np.round(vals, 4))          # [1. 3.]  (order may vary)
print("eigenvectors (columns):\n", np.round(vecs, 4))
 
# verify A x = λ x for the first eigenpair
lam, x = vals[0], vecs[:, 0]
print("A x =", np.round(A @ x, 4))
print("λ x =", np.round(lam * x, 4))
print("match:", np.allclose(A @ x, lam * x))
 
# symmetric matrices: use eigh for real eigenvalues + orthonormal eigenvectors
w, Q = np.linalg.eigh(A)
print("orthonormal eigenvectors:", np.allclose(Q.T @ Q, np.eye(2)))

eigenvalues: [1. 3.]
eigenvectors (columns):
 [[-0.7071  0.7071]
 [ 0.7071  0.7071]]
A x = [-0.7071  0.7071]
λ x = [-0.7071  0.7071]
match: True
orthonormal eigenvectors: True

eigenvalues: [1. 3.]
eigenvectors (columns):
 [[-0.7071  0.7071]
 [ 0.7071  0.7071]]
A x = [-0.7071  0.7071]
λ x = [-0.7071  0.7071]
match: True
orthonormal eigenvectors: True

Why this matters for ML

• PCA diagonalizes the covariance matrix; its eigenvectors are the principal directions, eigenvalues the variance along each.
• PageRank / spectral clustering read structure off the leading eigenvectors of a graph matrix.
• Optimization & stability: the eigenvalues of the Hessian tell you whether a critical point is a min, max, or saddle, and how fast gradient descent converges.

🧪 Try It Yourself

Exercise 1 – Compute eigenvalues and eigenvectors

Exercise 2 – Verify the eigenvalue equation

Exercise 3 – Symmetric ⇒ orthonormal eigenvectors

Recap

• An eigenvector is a direction a matrix only scales: $A\mathbf{x} = \lambda\mathbf{x}$; $\lambda$ is the eigenvalue.
• Eigenvalues are roots of the characteristic polynomial $\det(A - \lambda I) = 0$; eigenvectors span the eigenspace (null space of $A - \lambda I$).
• Defective matrices lack $n$ independent eigenvectors; symmetric matrices never do (spectral theorem: real eigenvalues, orthonormal eigenvectors).
• These drive PCA, PageRank, spectral clustering, and stability analysis.

Next: a specialized “square root” for symmetric positive-definite matrices — Cholesky Decomposition.