The curse of dimensionality

Real datasets often have dozens, hundreds, or thousands of features. More features should mean more information — but past a point they make training slower and harder, not better. This is the curse of dimensionality.

Two counterintuitive facts explain why:

• in a high-dimensional space, most points end up near the edge of the space instead of the middle — there’s no “safe” interior to be in
• two random points in a 1,000,000-dimensional cube are, on average, far apart — so every dataset feels sparse, and a new point is likely far from anything the model has seen before

The more dimensions a training set has, the more it risks overfitting, because the model is extrapolating across mostly empty space. In theory you could fix this by collecting more data, but the amount of data needed to keep points close together grows exponentially with the number of features — completely impractical past a few dozen dimensions.

Two ways to reduce dimensions: projection vs. manifold learning

Most real-world features aren’t spread out uniformly — many are almost constant, others are highly correlated. That means the data actually lives close to a much lower-dimensional subspace inside the high-dimensional one. There are two main strategies for finding it:

• Projection — squash the data straight down onto a flatter subspace (like flattening a tilted 3D pancake of points onto a 2D plane). This is what PCA does.
• Manifold learning — assume the data lies on a shape that’s bent or twisted through the high-dimensional space (like a Swiss roll). Simply projecting would squash different layers of the roll together; you need to unroll it instead. Algorithms like LLE and t-SNE do this — more on them in the next page.

  flowchart TD
  D["High-dimensional data"] --> Q{"Does it lie near a flat subspace?"}
  Q -->|"yes"| P["Projection (PCA)"]
  Q -->|"no, bent/twisted"| M["Manifold Learning"]
  P --> R["Lower-dimensional data"]
  M --> R

PCA intuition: preserving variance

Before you can project data onto a lower-dimensional plane, you have to pick the right plane. Imagine a 2D cloud of points and three candidate 1D axes to project it onto: one keeps the cloud spread out (high variance), another squashes almost everything into a single clump (low variance), and a third sits in between.

PCA always picks the axis that preserves the most variance — it’s also the axis that minimizes the squared distance between the original points and their projections. Losing the least variance means losing the least information.

PCA doesn’t stop at one axis. It finds a first principal component (PC1) along the direction of maximum variance, then a second (PC2) orthogonal to the first, that captures the largest amount of the remaining variance — and so on, until it has as many principal components as the dataset has dimensions. Under the hood, scikit-learn finds these axes using Singular Value Decomposition (SVD), and it automatically centers the data for you first (PCA requires the data to be centered around the origin).

Visualize it

Watch PCA actually search for the direction of maximum variance instead of just showing the answer: the amber line sweeps from an arbitrary starting angle and rotates into alignment with PC1, while a “variance captured” meter fills up as it converges. Once it locks on, every point grows a thin line straight out to its projection onto PC1 — a 1D “compression” of this 2D data. Click to reshuffle the cloud into a new shape/orientation:

Scikit-learn: fit_transform and explained_variance_ratio_

import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
 
iris = load_iris()
X = iris.data  # 150 flowers, 4 features each
 
pca = PCA(n_components=2)
X2 = pca.fit_transform(X)
 
print("Original shape:", X.shape)
print("Reduced shape:", X2.shape)
print("Explained variance ratio:", np.round(pca.explained_variance_ratio_, 3))
print("Total variance kept:", round(pca.explained_variance_ratio_.sum(), 3))

import numpy as np
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
 
iris = load_iris()
X = iris.data  # 150 flowers, 4 features each
 
pca = PCA(n_components=2)
X2 = pca.fit_transform(X)
 
print("Original shape:", X.shape)
print("Reduced shape:", X2.shape)
print("Explained variance ratio:", np.round(pca.explained_variance_ratio_, 3))
print("Total variance kept:", round(pca.explained_variance_ratio_.sum(), 3))

Original shape: (150, 4)
Reduced shape: (150, 2)
Explained variance ratio: [0.925 0.053]
Total variance kept: 0.978

Original shape: (150, 4)
Reduced shape: (150, 2)
Explained variance ratio: [0.925 0.053]
Total variance kept: 0.978

explained_variance_ratio_explained_variance_ratio_ tells you what fraction of the dataset’s total variance lies along each principal component. Here, the first axis alone carries 92.5% of the information in the 4 original iris features — and just two axes keep 97.8% of it.

Choosing the right number of dimensions

Instead of guessing a number of dimensions, pick the smallest number that keeps a target amount of variance (95% is a common choice) — unless you’re reducing to 2 or 3 dimensions purely for plotting.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
 
X = load_digits().data  # 1,797 images, 64 pixels each
print("Original shape:", X.shape)
 
pca = PCA()
pca.fit(X)
cumsum = np.cumsum(pca.explained_variance_ratio_)
d = int(np.argmax(cumsum >= 0.95) + 1)
print("Dimensions needed for 95% variance:", d)
 
# Equivalent, one-step version: pass a variance ratio instead of a count.
pca95 = PCA(n_components=0.95)
X_reduced = pca95.fit_transform(X)
print("Reduced shape:", X_reduced.shape)

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
 
X = load_digits().data  # 1,797 images, 64 pixels each
print("Original shape:", X.shape)
 
pca = PCA()
pca.fit(X)
cumsum = np.cumsum(pca.explained_variance_ratio_)
d = int(np.argmax(cumsum >= 0.95) + 1)
print("Dimensions needed for 95% variance:", d)
 
# Equivalent, one-step version: pass a variance ratio instead of a count.
pca95 = PCA(n_components=0.95)
X_reduced = pca95.fit_transform(X)
print("Reduced shape:", X_reduced.shape)

Original shape: (1797, 64)
Dimensions needed for 95% variance: 29
Reduced shape: (1797, 29)

Original shape: (1797, 64)
Dimensions needed for 95% variance: 29
Reduced shape: (1797, 29)

PCA(n_components=0.95)PCA(n_components=0.95) is the option to remember: pass a float between 0 and 1 and scikit-learn works out dd for you.

PCA for compression: inverse_transform

Because reducing dimensions is a linear projection, you can also run it backward — decompressing the reduced data back to the original number of dimensions with inverse_transform()inverse_transform(). You won’t get the exact original data back (the dropped variance is gone for good), but you’ll get something close. The average squared gap between the original and the reconstruction is called the reconstruction error.

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
import numpy as np
 
X = load_digits().data
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)
X_recovered = pca.inverse_transform(X_reduced)
 
mse = np.mean((X - X_recovered) ** 2)
print("Reconstruction MSE:", round(mse, 2))

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
import numpy as np
 
X = load_digits().data
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)
X_recovered = pca.inverse_transform(X_reduced)
 
mse = np.mean((X - X_recovered) ** 2)
print("Reconstruction MSE:", round(mse, 2))

Reconstruction MSE: 0.85

Reconstruction MSE: 0.85

Randomized PCA and Incremental PCA

Full SVD gets slow as the number of features grows. Two variants keep PCA practical at scale:

• Randomized PCA (svd_solver="randomized"svd_solver="randomized") — a stochastic shortcut that approximates the first dd components dramatically faster than full SVD when dd is much smaller than the number of features. It’s the default whenever scikit-learn decides the dataset is large enough to benefit.
• Incremental PCA — full PCA needs the entire training set in memory at once. IncrementalPCAIncrementalPCA instead consumes the data one mini-batch at a time via partial_fit()partial_fit(), so it works for datasets (or streams) too big to fit in RAM.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA, IncrementalPCA
 
X = load_digits().data
 
pca_full = PCA(n_components=29, svd_solver="full").fit(X)
pca_rand = PCA(n_components=29, svd_solver="randomized", random_state=42).fit(X)
print("Full SVD variance kept:      ", round(pca_full.explained_variance_ratio_.sum(), 3))
print("Randomized SVD variance kept:", round(pca_rand.explained_variance_ratio_.sum(), 3))
 
inc_pca = IncrementalPCA(n_components=29)
for X_batch in np.array_split(X, 10):     # feed 10 mini-batches
    inc_pca.partial_fit(X_batch)
X_reduced = inc_pca.transform(X)
print("Incremental PCA reduced shape:", X_reduced.shape)

import numpy as np
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA, IncrementalPCA
 
X = load_digits().data
 
pca_full = PCA(n_components=29, svd_solver="full").fit(X)
pca_rand = PCA(n_components=29, svd_solver="randomized", random_state=42).fit(X)
print("Full SVD variance kept:      ", round(pca_full.explained_variance_ratio_.sum(), 3))
print("Randomized SVD variance kept:", round(pca_rand.explained_variance_ratio_.sum(), 3))
 
inc_pca = IncrementalPCA(n_components=29)
for X_batch in np.array_split(X, 10):     # feed 10 mini-batches
    inc_pca.partial_fit(X_batch)
X_reduced = inc_pca.transform(X)
print("Incremental PCA reduced shape:", X_reduced.shape)

Full SVD variance kept:       0.955
Randomized SVD variance kept: 0.955
Incremental PCA reduced shape: (1797, 29)

Full SVD variance kept:       0.955
Randomized SVD variance kept: 0.955
Incremental PCA reduced shape: (1797, 29)

Randomized and full SVD land on almost the same explained variance here — the randomized version just gets there faster on bigger datasets.

A peek at Kernel PCA

Ordinary PCA only draws straight axes through the data. Kernel PCA borrows the “kernel trick” from SVMs (see Phase 5) to perform a nonlinear projection instead — useful when clusters are separable, but not by any straight line.

from sklearn.decomposition import KernelPCA
from sklearn.datasets import load_digits
 
X = load_digits().data
rbf_pca = KernelPCA(n_components=2, kernel="rbf", gamma=0.001, random_state=42)
X_reduced = rbf_pca.fit_transform(X)
print("Kernel PCA reduced shape:", X_reduced.shape)

from sklearn.decomposition import KernelPCA
from sklearn.datasets import load_digits
 
X = load_digits().data
rbf_pca = KernelPCA(n_components=2, kernel="rbf", gamma=0.001, random_state=42)
X_reduced = rbf_pca.fit_transform(X)
print("Kernel PCA reduced shape:", X_reduced.shape)

Kernel PCA reduced shape: (1797, 2)

Kernel PCA reduced shape: (1797, 2)

Because Kernel PCA is unsupervised, there’s no obvious accuracy score to pick the best kernel/gamma — Géron suggests either grid-searching a downstream classifier’s accuracy, or choosing the kernel that minimizes the reconstruction pre-image error (only available if you set fit_inverse_transform=Truefit_inverse_transform=True).

🧪 Try It Yourself

Exercise 1 – Reduce Iris to Two Dimensions

Exercise 2 – Read the Explained Variance Ratio

Exercise 3 – Reconstruct with inverse_transform

Next

Continue to t-SNE and Manifold Learning — PCA only draws straight axes through the data; next you’ll see the nonlinear techniques built specifically for visualizing high-dimensional clusters in 2D.