Every phase so far had an answer key. You fit a model, compared its predictions against yy, and got a number that told you whether you were right.

This phase has no yy. That removes the labelling cost and, with it, the ability to check your work. Which makes the central question of unsupervised learning not “what does the algorithm find?” but “how would I know if it were wrong?” — and every page here takes that question seriously enough to include at least one measurement where the method fails.

What this phase covers

Two halves. The first four pages group rows; the last three reduce or mine columns.

  flowchart TD
  X["Unlabelled X"] --> R{"Group rows, or reduce columns?"}
  R -->|"rows"| C["Clustering"]
  R -->|"columns"| D["Dimensionality reduction"]
  R -->|"rare rows"| A["Anomaly detection"]
  R -->|"co-occurring items"| AR["Association rules"]
  C --> C1["k-means — round clusters, need k"]
  C --> C2["Agglomerative — full hierarchy, cut later"]
  C --> C3["DBSCAN — any shape, marks noise"]
  D --> D1["PCA — linear, invertible, reusable"]
  D --> D2["t-SNE / LLE — curved, visualisation only"]
  A --> A1["Isolation Forest"]
  AR --> AR1["Apriori / FP-Growth"]

The one thing that decides everything

Every algorithm in this phase — clustering, anomaly detection and dimensionality reduction alike — operates on distances or variances in your feature space. So the feature space is the model.

None of these is a bug. In each case the algorithm faithfully reported the structure implied by the distances it was given. Get the feature space right and the algorithm barely matters. Get it wrong and no algorithm can save you.

How to check unsupervised work

There is no accuracy score, but there is more than nothing:

That last row is not a joke. k-means on uniform noise returns three clusters with a silhouette of 0.378 — a number you would happily ship. The shuffled-null comparison and your own eyes are what catch it.

Before you start

• Feature scaling — load-bearing for every page here.
• Transformation pipelines — the moment a cluster label becomes a feature, the leakage rules apply as usual.
• Basic linear algebra — eigenvectors and eigenvalues for the PCA page. The page derives what it needs, but familiarity helps.
• Decision trees — the Isolation Forest page assumes you know what a tree split is.

What you’ll be able to do afterwards

1. Choose a distance metric deliberately and explain what it makes “similar”.
2. Derive the k-means update rules from the inertia objective and prove the algorithm terminates.
3. Compute a silhouette coefficient, all four linkage distances, and a lift, by hand.
4. Read a dendrogram and pick a cut height from the merge-distance plateau.
5. Choose DBSCAN’s epseps from the k-distance knee rather than by trial and error.
6. Explain why PCA’s components are eigenvectors of the covariance matrix.
7. Say precisely what a t-SNE plot does and does not license you to conclude.
8. Recognise, in each case, the geometry that breaks the method you are using.

How long it takes

Practice project

Segment one dataset four ways, and defend a choice. Pick anything with 2,000+ rows and at least eight numeric features — customer transactions, country-level indicators, sensor readings, song audio features.

Part 1 — Prepare. Scale the features. Run PCA and record how many components reach 90% of the variance. Note whether that number surprises you.

Part 2 — Cluster. Fill in this table on the scaled data:

Part 3 — Visualise. Plot the PCA projection and a t-SNE embedding, coloured by each clustering. Write one sentence on what the t-SNE plot licenses you to say, and one on what it does not.

Part 4 — Validate. Shuffle each column independently, rerun your best method, and compare the silhouette against the real one. If they are close, you have found nothing.

Part 5 — Decide. In writing:

1. Which clustering would you ship, and why?
2. What does each cluster mean? Name them, using the feature means per cluster.
3. Which points did DBSCAN call noise? Look at ten of them. Are they errors, outliers, or a real small group?

If Part 4 says your structure is not distinguishable from noise, that is the finding. Report it. It is a more valuable result than a segmentation nobody should act on.

Next

Start with Introduction to Clustering — what a cluster is, why the distance metric decides the answer before the algorithm runs, and how to score a grouping when there is nothing to check it against.