What you’ll learn

• how classification differs from regression
• why a classifier’s probability output needs a threshold to become a label
• what a decision boundary is, visually
• why accuracy can be a misleading metric on skewed data
• binary vs multiclass classification, and a preview of OvR/OvO

Regression vs classification

• Regression: predict a number (a house price, a temperature)
• Classification: predict a label (spam/not spam, digit 0-9)

Many classifiers don’t jump straight to a label — they first estimate a probability:

• P(class = 1 | X)P(class = 1 | X)

…then apply a threshold (0.5 by default) to decide the final label.

Hands-On Machine Learning opens its classification chapter with the MNIST dataset — 70,000 images of handwritten digits, each labeled 0-9. It’s called the “hello world” of classification because almost every new algorithm gets tried on it first. We’ll use a smaller, offline stand-in — scikit-learn’s built-in load_digits()load_digits() (1,797 images) — so every example below runs without downloading anything.

  flowchart LR
  X["Features (pixels)"] --> M["Classifier"]
  M --> P["Probability score"]
  P --> T["Threshold (e.g. 0.5)"]
  T --> Y["Predicted label"]

A binary classifier: “is this digit a 5?”

The book’s first model is a “5-detector” — a binary classifier that only asks is this a 5, or not? Here’s the same idea using load_digits()load_digits() and an SGDClassifierSGDClassifier:

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import accuracy_score
 
digits = load_digits()
X, y = digits.data, digits.target
 
# Binary target: True for every 5, False for everything else
y_is_five = (y == 5)
 
X_train, X_test, y_train, y_test = train_test_split(
    X, y_is_five, test_size=0.2, random_state=42
)
 
sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train)
 
preds = sgd_clf.predict(X_test)
print("Accuracy:", round(accuracy_score(y_test, preds), 4))
# Accuracy: 0.9861

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import accuracy_score
 
digits = load_digits()
X, y = digits.data, digits.target
 
# Binary target: True for every 5, False for everything else
y_is_five = (y == 5)
 
X_train, X_test, y_train, y_test = train_test_split(
    X, y_is_five, test_size=0.2, random_state=42
)
 
sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train)
 
preds = sgd_clf.predict(X_test)
print("Accuracy:", round(accuracy_score(y_test, preds), 4))
# Accuracy: 0.9861

Why accuracy alone can lie to you

Only about 10% of the digits are actually 5s. Compare the classifier above to a “dumb” model that always predicts “not a 5”:

from sklearn.dummy import DummyClassifier
 
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_preds = dummy.predict(X_test)
print("Dummy accuracy:", round(accuracy_score(y_test, dummy_preds), 4))
# Dummy accuracy: 0.8694

from sklearn.dummy import DummyClassifier
 
dummy = DummyClassifier(strategy="most_frequent")
dummy.fit(X_train, y_train)
dummy_preds = dummy.predict(X_test)
print("Dummy accuracy:", round(accuracy_score(y_test, dummy_preds), 4))
# Dummy accuracy: 0.8694

The dummy classifier — which never detects a single real 5 — still scores 86.9% accuracy, just because 5s are rare. This is why accuracy is generally not the preferred metric for classifiers, especially on skewed (imbalanced) datasets. Later pages in this phase cover the confusion matrix, precision, recall, and ROC/AUC — the metrics built to catch exactly this trap.

Decision boundary intuition

A classifier learns a boundary through feature space that separates one class from another. Points on one side get one label; points on the other side get the other label. Where that boundary sits — and how it curves — is what differs between logistic regression (a straight line), KNN (a jagged neighborhood-based boundary), SVM (a max-margin boundary), and decision trees (axis-aligned rectangles).

Binary vs multiclass

• binary: 2 classes (0/1) — e.g. “is this a 5?”
• multiclass: 3+ classes — e.g. “which digit, 0-9?”

Some algorithms (SGD, Random Forest, Naïve Bayes) handle multiple classes natively. Others (Logistic Regression, SVM) are strictly binary, but scikit-learn can automatically combine several binary classifiers using:

• One-vs-Rest (OvR): train one classifier per class, pick the highest score
• One-vs-One (OvO): train a classifier for every pair of classes, most wins decide

You’ll see both strategies again in the Logistic Regression and SVM pages.

Common pitfalls

• class imbalance (see the dummy-classifier trap above)
• picking the wrong metric for the business problem
• data leakage via preprocessing (e.g. scaling before splitting)

Mini-checkpoint

Pick a real classification problem you know (spam, fraud, churn) and write down:

• the classes (labels)
• the cost of a false negative vs a false positive
• the best metric for that situation, and why

🧪 Try It Yourself

Exercise 1 – Build a Binary Target

Exercise 2 – Fit an SGDClassifier

Exercise 3 – Compare Against a Dummy Baseline

Next

Continue to Logistic Regression (Binary vs Multiclass) to see the first real classification algorithm — including the sigmoid function, log loss, and Softmax for more than two classes.