When a straight line isn’t enough

PCA reduces dimensions by projecting data onto flat axes. That works great when the data really does lie close to a flat subspace — but not every dataset is flat. The classic counterexample from the book is the Swiss roll: a 2D sheet of points rolled up into a spiral in 3D. Squash it flat by dropping one axis (a projection) and you flatten different layers of the roll on top of each other, destroying the structure. What you actually want is to unroll it.

The manifold hypothesis

A d-dimensional manifold is a shape that, zoomed in closely enough, looks like a flat d-dimensional surface — even though it’s bent or twisted through a higher-dimensional space. The Swiss roll is a 2D manifold living in 3D: locally it’s flat, globally it’s rolled up.

The manifold hypothesis says most real-world high-dimensional datasets actually lie close to a much lower-dimensional manifold. Think about handwritten digits: every valid digit image has connected strokes, white borders, and roughly centered content. Out of every possible way to color a grid of pixels, only a tiny sliver looks like a real digit — so the “digit manifold” has far fewer effective degrees of freedom than the raw pixel count suggests. Manifold learning algorithms try to model that lower-dimensional shape directly, instead of just projecting onto a flat plane.

  flowchart TD
  D["High-dimensional data"] --> Q{"Does it lie near a flat subspace?"}
  Q -->|"yes"| P["Projection: PCA"]
  Q -->|"no — bent / twisted"| M["Manifold Learning"]
  M --> TS["t-SNE (for visualization)"]
  M --> L["LLE (unroll local neighborhoods)"]

t-SNE: built for visualization

t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality by trying to keep similar instances close together and dissimilar instances far apart in the new, low-dimensional space. Unlike PCA, it makes no attempt to preserve straight-line distances or overall variance — its entire job is to produce a 2D (or 3D) picture where clusters that exist in high-dimensional space are still visibly separated after the squeeze.

That focus is also its biggest limitation: t-SNE is a visualization tool, not a general-purpose preprocessing step. A few practical consequences:

• it’s normally run with n_componentsn_components set to 2 or 3 — just enough to plot
• TSNETSNE has no transform()transform() method — you can’t fit it once and reuse it on new points the way you can with PCAPCA; every call is fit_transformfit_transform
• it’s relatively slow and doesn’t scale to huge datasets the way PCA does

Perplexity: t-SNE’s main knob

t-SNE’s key hyperparameter is perplexity, roughly “how many neighbors each point should try to stay close to.” Small values focus on tight local structure; larger values consider a broader neighborhood. There’s no single correct value — Géron’s own advice for most datasets is to try a few values, typically somewhere between 5 and 50, and see which produces the most visually sensible clusters.

Visualize it

You can’t literally draw a 64-dimensional digit image moving through space — but this is what t-SNE is doing to it: pulling points that are alike (same color = same hidden group) toward each other, and letting dissimilar points drift apart, until the high-dimensional groups collapse into clean 2D clusters. Click to reshuffle and re-run the collapse:

t-SNE in scikit-learn

from sklearn.datasets import load_digits
from sklearn.manifold import TSNE
 
digits = load_digits()
X = digits.data[:80]   # keep it small so it fits/trains fast
 
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init="pca")
X2 = tsne.fit_transform(X)
 
print("Original shape:", X.shape)
print("Embedded shape:", X2.shape)

from sklearn.datasets import load_digits
from sklearn.manifold import TSNE
 
digits = load_digits()
X = digits.data[:80]   # keep it small so it fits/trains fast
 
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init="pca")
X2 = tsne.fit_transform(X)
 
print("Original shape:", X.shape)
print("Embedded shape:", X2.shape)

Original shape: (80, 64)
Embedded shape: (80, 2)

Original shape: (80, 64)
Embedded shape: (80, 2)

Does t-SNE actually keep similar digits closer together? Compare the average distance between points that share a digit label to the average distance between points that don’t, after embedding:

import numpy as np
from sklearn.datasets import load_digits
from sklearn.manifold import TSNE
 
digits = load_digits()
X, y = digits.data[:80], digits.target[:80]
 
X2 = TSNE(n_components=2, perplexity=15, random_state=42, init="pca").fit_transform(X)
 
same, diff = [], []
for i in range(len(y)):
    for j in range(i + 1, len(y)):
        d = np.linalg.norm(X2[i] - X2[j])
        (same if y[i] == y[j] else diff).append(d)
 
print("Mean distance, same digit:     ", round(np.mean(same), 1))
print("Mean distance, different digit:", round(np.mean(diff), 1))
print("Same-digit points end up closer:", np.mean(same) < np.mean(diff))

import numpy as np
from sklearn.datasets import load_digits
from sklearn.manifold import TSNE
 
digits = load_digits()
X, y = digits.data[:80], digits.target[:80]
 
X2 = TSNE(n_components=2, perplexity=15, random_state=42, init="pca").fit_transform(X)
 
same, diff = [], []
for i in range(len(y)):
    for j in range(i + 1, len(y)):
        d = np.linalg.norm(X2[i] - X2[j])
        (same if y[i] == y[j] else diff).append(d)
 
print("Mean distance, same digit:     ", round(np.mean(same), 1))
print("Mean distance, different digit:", round(np.mean(diff), 1))
print("Same-digit points end up closer:", np.mean(same) < np.mean(diff))

Mean distance, same digit:      2.5
Mean distance, different digit: 12.1
Same-digit points end up closer: True

Mean distance, same digit:      2.5
Mean distance, different digit: 12.1
Same-digit points end up closer: True

Points sharing a digit end up roughly 5x closer on average than points of different digits — exactly the behavior t-SNE is designed to produce.

Locally Linear Embedding (LLE), briefly

LLE is another manifold-learning technique, but unlike PCA or t-SNE it never projects the data at all. Instead, for every point it:

1. finds that point’s kk nearest neighbors
2. figures out the weights that best reconstruct the point as a linear combination of those neighbors
3. finds a low-dimensional set of positions that preserves those same neighbor weights

Because it only cares about local neighborhoods, LLE is particularly good at unrolling twisted manifolds like the Swiss roll. The trade-off is that it scales poorly to very large datasets (its heaviest step grows with the square of the number of instances).

from sklearn.datasets import load_digits
from sklearn.manifold import LocallyLinearEmbedding
 
X = load_digits().data[:100]
 
lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10, random_state=42)
X2 = lle.fit_transform(X)
 
print("LLE embedded shape:", X2.shape)
print("Reconstruction error:", round(lle.reconstruction_error_, 6))

from sklearn.datasets import load_digits
from sklearn.manifold import LocallyLinearEmbedding
 
X = load_digits().data[:100]
 
lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10, random_state=42)
X2 = lle.fit_transform(X)
 
print("LLE embedded shape:", X2.shape)
print("Reconstruction error:", round(lle.reconstruction_error_, 6))

LLE embedded shape: (100, 2)
Reconstruction error: 0.000443

LLE embedded shape: (100, 2)
Reconstruction error: 0.000443

A tiny reconstruction_error_reconstruction_error_ means LLE found low-dimensional positions that preserve each point’s local neighborhood relationships almost exactly.

Mini-checkpoint

If you fit TSNETSNE on your training set to make a nice plot, can you reuse that same fitted object to embed a brand-new point that just arrived?

(No — TSNETSNE has no transform()transform() method. You’d need to refit on the combined dataset, or use a different technique such as PCA if you need to transform new data later.)

🧪 Try It Yourself

Exercise 1 – Embed Digits into 2D with t-SNE

Exercise 2 – Check LLE’s Reconstruction Error

Exercise 3 – t-SNE Has No transform() for New Data

Next

That wraps up Phase 6’s clustering, anomaly-detection, association-rule, and dimensionality-reduction techniques. Continue to Phase 7 with Underfitting vs. Overfitting — now that you can build and simplify models, it’s time to learn how to tell whether one is actually any good.