What you’ll learn

• the “widest street” intuition behind maximum-margin classification
• what a support vector actually is, and why most training points don’t matter
• hard margin vs soft margin, and what the CC hyperparameter controls
• how the kernel trick lets a linear algorithm draw curved boundaries
• why feature scaling isn’t optional for SVMs

Core idea: maximum margin

Géron’s book explains SVMs with a simple picture: imagine trying to separate two classes with a straight line. Many lines would technically work, but some come so close to the training points that they’d probably misclassify new data. An SVM classifier instead fits the widest possible street between the two classes — think of it as a road with the decision boundary as the center line, and the road’s edges pushed out as far as they can go before touching a point of either class. This is called large margin classification.

  flowchart LR
  A["Class -1 points"] --> M["Maximize margin width"]
  B["Class +1 points"] --> M
  M --> H["Max-margin hyperplane
(decision boundary)"]

Support vectors

Only the points sitting exactly on the edge of the street determine where that street ends up — these are the support vectors. Adding more training instances well outside the street (off the road entirely) doesn’t move the boundary at all, because the boundary is fully “supported” by just those edge points. This is very different from something like Logistic Regression, whose decision boundary shifts at least a little with every new point.

Hard margin vs soft margin

If you insist every single point stay off the street and on the correct side, that’s hard margin classification — but it only works on linearly separable data, and it’s very sensitive to outliers (one badly placed point can make a hard margin impossible to find, or force a much worse boundary).

Soft margin classification relaxes that: it looks for the best balance between a wide street and a small number of margin violations (points that end up on the street, or even on the wrong side of it). Scikit-learn’s CC hyperparameter controls that balance:

• low CC → more margin violations allowed, wider street, often generalizes better
• high CC → fewer violations allowed, narrower street, more prone to overfitting

Scaling matters

SVM tries to make the street as wide as possible in every feature’s units. If one feature ranges from 0–1 and another from 0–10,000, the “widest street” will end up nearly parallel to the small-range feature, ignoring the other one almost entirely. Always scale first:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
 
svm = Pipeline(
    steps=[
        ("scaler", StandardScaler()),
        ("model", SVC(kernel="rbf", C=1.0, gamma="scale", probability=True)),
    ]
)

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
 
svm = Pipeline(
    steps=[
        ("scaler", StandardScaler()),
        ("model", SVC(kernel="rbf", C=1.0, gamma="scale", probability=True)),
    ]
)

Linear SVM on the iris dataset

Géron’s book fits a linear SVM to detect Iris virginica, exactly like the Logistic Regression example from the previous page — but notice the loss function is different (hingehinge, not log loss):

import numpy as np
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
 
iris = datasets.load_iris()
X = iris["data"][:, (2, 3)]              # petal length, petal width
y = (iris["target"] == 2).astype(np.float64)  # Iris virginica
 
svm_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("linear_svc", LinearSVC(C=1, loss="hinge", dual=True, random_state=42)),
])
svm_clf.fit(X, y)
 
print(svm_clf.predict([[5.5, 1.7]]))
# [1.]

import numpy as np
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
 
iris = datasets.load_iris()
X = iris["data"][:, (2, 3)]              # petal length, petal width
y = (iris["target"] == 2).astype(np.float64)  # Iris virginica
 
svm_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("linear_svc", LinearSVC(C=1, loss="hinge", dual=True, random_state=42)),
])
svm_clf.fit(X, y)
 
print(svm_clf.predict([[5.5, 1.7]]))
# [1.]

Unlike Logistic Regression, an SVM classifier does not output class probabilities by default — SVCSVC can approximate them (probability=Trueprobability=True), but it’s an expensive extra step, not something the algorithm naturally produces.

Kernels (non-linear boundaries)

Many real datasets aren’t linearly separable at all. One fix is to manually add polynomial features until the data becomes separable — but that can explode into a huge number of columns at high degrees. SVM’s real superpower is the kernel trick: it produces the exact same result as if you’d added those extra features, without ever actually computing them.

Common kernels:

• linear — no transformation; fastest, try this first
• polynomial — degreedegree controls how curvy the boundary can get
• RBF (Gaussian radial basis function) — measures similarity to “landmark” points; works well in most cases when the training set isn’t huge

from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
 
X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
 
poly_kernel_svm_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5)),
])
poly_kernel_svm_clf.fit(X, y)
 
rbf_kernel_svm_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001)),
])
rbf_kernel_svm_clf.fit(X, y)

from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
 
X, y = make_moons(n_samples=100, noise=0.15, random_state=42)
 
poly_kernel_svm_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5)),
])
poly_kernel_svm_clf.fit(X, y)
 
rbf_kernel_svm_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001)),
])
rbf_kernel_svm_clf.fit(X, y)

gammagamma acts like a regularization knob for the RBF kernel: a high gammagamma narrows each point’s bell-shaped zone of influence, letting the boundary wiggle tightly around individual instances (risk of overfitting); a low gammagamma widens that zone, producing a smoother boundary (risk of underfitting). Increasing gammagamma typically increases the number of support vectors needed, since more points end up close enough to the boundary to matter.

Pros and cons

Pros:

• strong on small-to-medium, complex datasets
• effective in high-dimensional spaces, even when features outnumber samples
• kernel trick gives non-linear power without engineering features by hand

Cons:

• SVCSVC’s training time scales roughly between O(m² · n)O(m² · n) and O(m³ · n)O(m³ · n) — it gets painfully slow past tens of thousands of instances (LinearSVCLinearSVC scales much better, roughly O(m · n)O(m · n), but has no kernel trick)
• less interpretable than a Decision Tree
• doesn’t output probabilities natively

SVM Regression (SVR)

Everything so far has used SVMs for classification — but the same “widest street” machinery works for regression too, if you flip the objective around. Instead of fitting the widest street between two classes while limiting how many points land on it, SVM Regression tries to fit as many instances as possible on the street, while limiting how many fall off it (margin violations now mean points outside the street, not inside it).

The width of that street is controlled by a hyperparameter called epsilon (εε): it defines an epsilon-insensitive tube around the regression line. Any training instance inside the tube doesn’t affect the model’s predictions at all — adding more points inside the tube changes nothing, which is why the model is called ε-insensitive. Only points on the edge of, or outside, the tube act like support vectors and shape the fit.

  flowchart LR
  A["SVM classification:
widen the street BETWEEN classes"] --> C["Objective flips"]
  B["SVM regression:
fit the street AROUND the data"] --> C
  C --> D["epsilon controls tube width"]
  D --> E["Points inside the tube:
ignored (epsilon-insensitive)"]
  D --> F["Points on/outside the tube:
support vectors"]

• larger epsilonepsilon → a wider tube, fewer points end up as support vectors, a coarser fit
• smaller epsilonepsilon → a narrower tube, more points sit on or outside it and become support vectors, a tighter fit

Just like on the classification side, LinearSVRLinearSVR is the regression equivalent of LinearSVCLinearSVC (scales roughly linearly with training-set size, no kernel trick), while SVRSVR is the regression equivalent of SVCSVC (supports the kernel trick, but gets slow on large training sets). And exactly as before, scale your features first.

import numpy as np
from sklearn.svm import LinearSVR, SVR
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
 
rng = np.random.RandomState(42)
X = np.sort(5 * rng.rand(40, 1), axis=0)
y = (2 * X.ravel() + 3) + rng.normal(0, 0.5, X.shape[0])
 
# Linear SVM regression - a wide (epsilon=1.5) street around a straight line
svm_reg = Pipeline([
    ("scaler", StandardScaler()),
    ("linear_svr", LinearSVR(epsilon=1.5, random_state=42)),
])
svm_reg.fit(X, y)
print(round(svm_reg.predict([[3.0]])[0], 2))
# 7.97
 
# Kernelized SVM regression - handles curved (nonlinear) relationships too
svm_poly_reg = SVR(kernel="poly", degree=2, C=100, epsilon=0.1)
svm_poly_reg.fit(X, y)
print(round(svm_poly_reg.predict([[3.0]])[0], 2))
# 8.22

import numpy as np
from sklearn.svm import LinearSVR, SVR
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
 
rng = np.random.RandomState(42)
X = np.sort(5 * rng.rand(40, 1), axis=0)
y = (2 * X.ravel() + 3) + rng.normal(0, 0.5, X.shape[0])
 
# Linear SVM regression - a wide (epsilon=1.5) street around a straight line
svm_reg = Pipeline([
    ("scaler", StandardScaler()),
    ("linear_svr", LinearSVR(epsilon=1.5, random_state=42)),
])
svm_reg.fit(X, y)
print(round(svm_reg.predict([[3.0]])[0], 2))
# 7.97
 
# Kernelized SVM regression - handles curved (nonlinear) relationships too
svm_poly_reg = SVR(kernel="poly", degree=2, C=100, epsilon=0.1)
svm_poly_reg.fit(X, y)
print(round(svm_poly_reg.predict([[3.0]])[0], 2))
# 8.22

LinearSVR(epsilon=1.5)LinearSVR(epsilon=1.5) reproduces the book’s large-margin regression example; swapping in SVR(kernel="poly", ...)SVR(kernel="poly", ...) handles curved relationships the same way the kernel trick handles curved decision boundaries in classification — a large CC here means little regularization (fit the training data closely), while a smaller CC trades some fit for a smoother, more regularized line.

Mini-checkpoint

Try a linear kernel and an RBF kernel on the same dataset and compare their decision boundaries — the RBF one should curve around clusters that the linear one can only cut with a straight line.

🧪 Try It Yourself

Exercise 1 – Fit a Linear SVM

Exercise 2 – Count the Support Vectors

Exercise 3 – See Gamma’s Effect on the RBF Kernel

Exercise 4 – Widen the Epsilon Tube in SVR

Next

Continue to Decision Trees - Entropy and Gini Impurity for a completely different, rule-based approach to classification.