What you’ll learn

• the manifold hypothesis, and why linear projection cannot exploit it
• the t-SNE objective: Gaussian similarities in, Student-t similarities out, KL divergence between
• what perplexity actually controls, measured by the KL divergence it reaches
• the two things t-SNE plots do not preserve — cluster size and cluster distance — quantified
• trustworthiness as a number you can put on an embedding: PCA 0.830, t-SNE 0.992
• LLE and modified LLE, which unroll a Swiss roll that PCA merely flattens
• why t-SNE has no transformtransform, and what UMAP changes

Intuition

A $28 \times 28$ grayscale image lives in 784 dimensions. But the set of images that look like handwritten digits is a vanishingly small part of that space — almost every point in $\mathbb{R}^{784}$ is static. The digits occupy a thin, curved, low-dimensional sheet inside it.

That is the manifold hypothesis: real high-dimensional data concentrates near a low-dimensional manifold. If you could find the manifold’s own coordinates, you could describe each image with a handful of numbers instead of 784.

PCA assumes that manifold is flat. When it is not, PCA does the only thing a rotation can do: it photographs the curved surface from the best available angle, and everything on the far side lands on top of everything on the near side.

Manifold learning drops the flatness assumption. The methods differ in what they preserve instead:

• LLE preserves each point’s local linear relationship with its neighbours.
• Isomap preserves geodesic distances measured along the manifold.
• t-SNE preserves neighbourhood membership, and explicitly gives up on everything else.

  flowchart TD
  D["High-dimensional data"] --> Q{"Does it lie near a FLAT subspace?"}
  Q -->|"yes"| P["PCA — fast, invertible, preserves global structure"]
  Q -->|"no, it is curved"| M["Manifold learning"]
  M --> L["LLE / Isomap
reconstruct the manifold's coordinates"]
  M --> T["t-SNE / UMAP
preserve neighbourhoods for VISUALISATION"]
  T --> W["Global distances are NOT meaningful"]

The Swiss roll

The standard demonstration is a 2-D sheet rolled up in 3-D. It is genuinely two-dimensional — every point has a position along the roll and a position across it — but the rolling makes points that are far apart along the sheet end up close together in space.

figureOne manifold, three attempts to flatten itmatplotlib

Trustworthiness (defined below) measures whether points that are neighbours in the embedding were also neighbours in the original space. Every method scores well because that is a local question — and it is precisely the question t-SNE optimises.

The math

t-SNE builds two probability distributions over pairs of points and makes them match.

In the high-dimensional space, define a conditional probability that $i$ would pick $j$ as its neighbour, using a Gaussian centred on $i$:

Each point gets its own bandwidth $\sigma_i$, which is what lets t-SNE handle regions of different density. Symmetrise:

In the low-dimensional space, use a Student-t distribution with one degree of freedom — a Cauchy — instead of a Gaussian:

Minimise the Kullback-Leibler divergence between them by gradient descent on the $\mathbf{y}_i$:

Why the heavy tail

That Student-t in the output is the “t” in t-SNE, and it exists to solve the crowding problem.

In $p$ dimensions you can place many more points at mutually moderate distances than you can in 2. The volume of a ball of radius $r$ grows like $r^p$, so a neighbourhood that comfortably holds 1,000 points in 50-D has nowhere to put them in 2-D. If both distributions were Gaussian, the only way to fit everyone in would be to crush all the moderately-distant pairs into the centre.

The Cauchy’s heavy tail means a large output distance still produces a non-negligible $q_{ij}$. So a moderately-dissimilar pair can be placed far apart without incurring a big penalty, which frees up room in the middle for the genuinely-near pairs. Distant clusters get pushed apart and gaps open up.

Why KL divergence is asymmetric — and what that costs

$\mathrm{KL}(P \,\|\, Q)$ is weighted by $p_{ij}$. Look at the two error modes:

• Large $p_{ij}$, small $q_{ij}$ (near in high-D, far in the map): the term $p_{ij}\log(p_{ij}/q_{ij})$ is large. Heavily penalised.
• Small $p_{ij}$, large $q_{ij}$ (far in high-D, near in the map): $p_{ij}$ is tiny, so the whole term is tiny. Barely penalised.

t-SNE therefore works hard to keep neighbours together and hardly cares about keeping non-neighbours apart. Local structure is faithful; global structure is not constrained at all. That single asymmetry explains every caveat on this page.

Perplexity

The bandwidth $\sigma_i$ is not set directly. Instead you set a target perplexity, and t-SNE binary-searches each $\sigma_i$ until the conditional distribution $P_i$ has that perplexity:

Perplexity is a smooth measure of “how many neighbours does this point effectively have?” A perplexity of 30 means each point’s Gaussian is widened or narrowed until it spreads its probability over roughly 30 others.

Perplexity, measured

On 800 digit images:

The KL divergence falls monotonically with perplexity — which is exactly why you must not choose perplexity by minimising KL. A larger perplexity spreads each $P_i$ over more points, making it a smoother target that a 2-D layout can match more easily. The objective is not comparable across perplexities.

figureThe same 800 digits, four perplexitiesmatplotlib

Practical guidance:

• The usual range is 5 to 50; 30 is the default and a good first try.
• Perplexity must be less than $n$; scikit-learn requires perplexity < n_samplesperplexity < n_samples.
• Run several and look for structure that survives. A cluster that appears at perplexity 5, 30 and 50 is probably real. One that appears only at perplexity 2 is probably not.

What the plot does not tell you

This is the most important section on the page.

Take three groups whose true geometry is known: a tight blob at the origin, a blob five times wider centred 6 units away, and another tight blob 40 units away. Run t-SNE at perplexity 30 and measure what came out:

A 5.6-fold difference in true size became a 1.1-fold difference in the plot. And the distances:

Two gaps differing by more than five-fold were rendered essentially identical.

figureWhat t-SNE equalisesmatplotlib

Three rules follow, and they are not negotiable:

1. Cluster sizes in a t-SNE plot are meaningless. Per-point bandwidths deliberately equalise density, so a sparse cluster and a dense one come out the same size.
2. Distances between clusters are meaningless. The KL asymmetry barely penalises far-in-high-D pairs, so their spacing is essentially unconstrained.
3. Apparent clusters may not be real. At low perplexity t-SNE produces crisp-looking clumps in uniform noise. Always check across perplexities.

What a t-SNE plot does tell you, reliably: which points are near which other points.

Trustworthiness: a number instead of a vibe

sklearn.manifold.trustworthinesssklearn.manifold.trustworthiness asks whether the $k$ nearest neighbours in the embedding were also near in the original space:

where $U_i^{(k)}$ are the points in $i$’s embedded neighbourhood that were not in its original neighbourhood, and $r(i,j)$ is $j$’s rank by distance from $i$ in the original space. It runs from 0 to 1, and it penalises false neighbours — pairs the map claims are close that were not.

On the full digits dataset:

Two dimensions of t-SNE classify digits better than the original 64 dimensions. That is not because t-SNE created information — it is the curse of dimensionality working in reverse: nearest neighbours are more meaningful in 2-D when the 2-D layout was built specifically to preserve neighbourhoods.

Do not conclude that t-SNE is a good preprocessing step for classification. That 0.9761 was computed on an embedding fitted to all the data, including the held-out folds. t-SNE has no transformtransform method, so you cannot embed a test set without refitting — which is exactly why this number is a diagnostic, not a pipeline.

figureTwo 2-D views of the same 64-D digitsmatplotlib

See it move

The next sketch is the crowding problem itself — the reason for the Student-t tail.

At $d = 4$ a Gaussian gives $1.1 \times 10^{-7}$ and the Student-t gives $0.0588$ — over five hundred thousand times more. That difference is the space t-SNE uses to separate clusters instead of piling them on top of each other.

In code

from sklearn.datasets import load_digits
from sklearn.manifold import TSNE, trustworthiness
 
X, y = load_digits(return_X_y=True)
 
ts = TSNE(
    n_components=2,        # 2 or 3; anything higher is not what the method is for
    perplexity=30,         # effective neighbourhood size; try 5, 30, 50
    init="pca",            # far more stable than "random"; the default since 1.2
    learning_rate="auto",  # max(n / early_exaggeration / 4, 50)
    random_state=0,
)
Z = ts.fit_transform(X)
 
print("KL divergence  ", round(ts.kl_divergence_, 4))
print("iterations     ", ts.n_iter_)
print("trustworthiness", round(trustworthiness(X, Z, n_neighbors=12), 4))   # 0.9917

from sklearn.datasets import load_digits
from sklearn.manifold import TSNE, trustworthiness
 
X, y = load_digits(return_X_y=True)
 
ts = TSNE(
    n_components=2,        # 2 or 3; anything higher is not what the method is for
    perplexity=30,         # effective neighbourhood size; try 5, 30, 50
    init="pca",            # far more stable than "random"; the default since 1.2
    learning_rate="auto",  # max(n / early_exaggeration / 4, 50)
    random_state=0,
)
Z = ts.fit_transform(X)
 
print("KL divergence  ", round(ts.kl_divergence_, 4))
print("iterations     ", ts.n_iter_)
print("trustworthiness", round(trustworthiness(X, Z, n_neighbors=12), 4))   # 0.9917

Note fit_transformfit_transform — there is no fitfit followed by transformtransform on new data. t-SNE optimises the positions of these points; a new point has no defined position without rerunning the whole optimisation.

Three parameters worth knowing:

init="pca"init="pca" starts the optimisation from the PCA projection. It is more reproducible than random initialisation and preserves a little global structure, which is close to free.

early_exaggerationearly_exaggeration (default 12) multiplies all $p_{ij}$ during the first 250 iterations, which forms tight clusters with wide gaps before the fine layout begins. Raise it if clusters are not separating.

learning_ratelearning_rate — the classic default of 200 is too low for large $n$ and produces a compressed “ball” with everything crushed together. "auto""auto" scales it with $n$ and is what you should use.

Preprocess with PCA first

For data with hundreds of features, reduce to about 50 with PCA before t-SNE. This is the standard recipe and it does three things: cuts the $O(n^2)$ distance computation, suppresses noise dimensions, and usually improves the embedding.

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.pipeline import make_pipeline
 
# The standard recipe for wide data.
embed = make_pipeline(
    PCA(n_components=50, random_state=0),
    TSNE(n_components=2, perplexity=30, init="pca", random_state=0),
)
Z = embed.fit_transform(X)      # fit_transform only — there is no transform

from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.pipeline import make_pipeline
 
# The standard recipe for wide data.
embed = make_pipeline(
    PCA(n_components=50, random_state=0),
    TSNE(n_components=2, perplexity=30, init="pca", random_state=0),
)
Z = embed.fit_transform(X)      # fit_transform only — there is no transform

Locally Linear Embedding

LLE takes a different route with no probabilities at all:

1. For each point, find its $k$ nearest neighbours.
2. Solve for the weights $W_{ij}$ that best reconstruct $\mathbf{x}_i$ as a linear combination of those neighbours, subject to $\sum_j W_{ij} = 1$.
3. Find low-dimensional $\mathbf{y}_i$ that are reconstructed by the same weights.

Step 2 captures the local geometry, and because the weights are constrained to sum to 1 they are invariant to rotation, rescaling and translation. Step 3 then asks: what layout in 2-D has the same local geometry?

from sklearn.datasets import make_swiss_roll
from sklearn.manifold import LocallyLinearEmbedding, trustworthiness
 
X, t = make_swiss_roll(n_samples=1200, noise=0.05, random_state=42)
 
std = LocallyLinearEmbedding(n_neighbors=12, n_components=2, random_state=0)
mod = LocallyLinearEmbedding(n_neighbors=12, n_components=2,
                             method="modified", random_state=0)
 
for name, model in (("standard", std), ("modified", mod)):
    Z = model.fit_transform(X)
    print(f"{name:9s} reconstruction_error_ {model.reconstruction_error_:.3e}"
          f"  trustworthiness {trustworthiness(X, Z, n_neighbors=12):.4f}")
 
# standard  reconstruction_error_ 1.073e-07  trustworthiness 0.9953
# modified  reconstruction_error_ 2.098e-06  trustworthiness 0.9942

from sklearn.datasets import make_swiss_roll
from sklearn.manifold import LocallyLinearEmbedding, trustworthiness
 
X, t = make_swiss_roll(n_samples=1200, noise=0.05, random_state=42)
 
std = LocallyLinearEmbedding(n_neighbors=12, n_components=2, random_state=0)
mod = LocallyLinearEmbedding(n_neighbors=12, n_components=2,
                             method="modified", random_state=0)
 
for name, model in (("standard", std), ("modified", mod)):
    Z = model.fit_transform(X)
    print(f"{name:9s} reconstruction_error_ {model.reconstruction_error_:.3e}"
          f"  trustworthiness {trustworthiness(X, Z, n_neighbors=12):.4f}")
 
# standard  reconstruction_error_ 1.073e-07  trustworthiness 0.9953
# modified  reconstruction_error_ 2.098e-06  trustworthiness 0.9942

Both score well on trustworthiness, but they look very different: standard LLE pinches the rectangle at one end, while modified LLE (which uses several weight vectors per neighbourhood) produces the clean rectangle in the figure above. Unlike t-SNE, LLE does have a transformtransform method — the weights generalise.

UMAP

UMAP is the modern alternative. It optimises a similar neighbourhood-preservation objective from a topological rather than probabilistic derivation, and in practice:

• It is much faster — near-linear in $n$ against t-SNE’s $O(n \log n)$ with a larger constant.
• It preserves more global structure, so inter-cluster distances mean somewhat more (though still not enough to quantify).
• It has a transformtransform method, so it can be used in a pipeline.

It is not in scikit-learn (pip install umap-learnpip install umap-learn) and its n_neighborsn_neighbors plays the role of perplexity. Its plots share the same caveats as t-SNE’s, just less severely.

Pitfalls

Measuring anything off the plot. Cluster radii were equalised from a 5.6× ratio to 1.1×, and gaps of 6.04 and 34.01 both came out near 39. Neither size nor distance survives.

Choosing perplexity by minimising KL divergence. It falls monotonically with perplexity — 0.6316 at 2, 0.4456 at 100 — because larger perplexity is an easier target. It is not a model selection criterion.

Believing clusters that appear at one perplexity. Low perplexity manufactures crisp clumps out of noise. Run 5, 30 and 50 and keep only what persists.

Trying to use it in a pipeline. TSNETSNE has fit_transformfit_transform and no transformtransform. Any cross-validation score computed on a t-SNE embedding of the full dataset — including the 0.9761 quoted above — has seen the held-out folds.

Leaving learning_rate=200learning_rate=200 on large data. On tens of thousands of points this produces a single dense ball. "auto""auto" is the fix.

Running it on raw high-dimensional data. Reduce to about 50 dimensions with PCA first. Faster, less noisy, usually a better result.

Comparing two t-SNE runs. Different seeds give different layouts of the same structure. Two plots are not comparable point-by-point; only the neighbourhood structure is.

Treating it as an alternative to PCA. They do different jobs. PCA is a reusable, invertible transform that preserves global geometry. t-SNE is a one-shot picture that preserves neighbourhoods.

Compare

🧪 Try It Yourself

Exercise 1 – PCA cannot unroll a Swiss roll

Exercise 2 – The perplexity sweep

Exercise 3 – Measure what the plot destroys

Exercise 4 – Distances between clusters are erased too

Exercise 5 – Trustworthiness against neighbour accuracy

Recap

• The manifold hypothesis: real high-dimensional data lies near a low-dimensional surface. PCA assumes that surface is flat; manifold learning does not.
• t-SNE matches Gaussian similarities in high-D against Student-t similarities in 2-D by minimising KL divergence. The heavy tail solves the crowding problem — at distance 4 it is over 500,000 times larger than a Gaussian.
• The KL asymmetry penalises splitting up neighbours heavily and barely penalises putting non-neighbours together, so local structure is faithful and global structure is not.
• Cluster sizes are meaningless: radii of 0.383, 1.941 and 0.345 all came out near 6.
• Cluster distances are meaningless: gaps of 6.04 and 34.01 both came out near 39.
• Perplexity is the effective neighbourhood size. KL falls monotonically with it (0.6316 → 0.4456), so it cannot be tuned that way — run several and keep what persists.
• Trustworthiness quantifies neighbourhood preservation: PCA 0.830, t-SNE 0.992 on digits.
• t-SNE has no transformtransform. LLE and UMAP do.

Exercise 6 – Check whether the distances survived

Next

That completes Phase 6. Go back to the phase overview for the practice project, or move on to Phase 7 - Model Optimization & Tuning, where every one of these choices — $k$, epseps, perplexity, the number of components — becomes something you select with cross-validation rather than by eye.