Before we can talk about distance, angle, or “closeness,” we need to measure how big a vector is. That measurement is a norm, written $\lVert \mathbf{x} \rVert$. Surprisingly, there isn’t just one correct ruler — and the choice of ruler shows up directly in machine learning as the difference between Lasso and Ridge regularization.

A real-life example: getting across a city

You’re at one corner of a grid-like city and want to reach a point 3 blocks east and 4 blocks north. Two honest answers to “how far is it?“:

• As the crow flies (Euclidean): $\sqrt{3^2 + 4^2} = 5$ blocks.
• As a taxi drives (Manhattan, following streets): $3 + 4 = 7$ blocks.

Both are legitimate lengths. Which one is “right” depends on how you’re allowed to move — and that is exactly what choosing a norm decides.

Definition

A norm on a vector space $V$ is a function $\lVert \cdot \rVert : V \to \mathbb{R}$ assigning each vector a length $\lVert \mathbf{x} \rVert$, satisfying, for all $\lambda \in \mathbb{R}$ and $\mathbf{x}, \mathbf{y} \in V$:

• Absolutely homogeneous: $\lVert \lambda\mathbf{x} \rVert = |\lambda|\,\lVert \mathbf{x} \rVert$
• Triangle inequality: $\lVert \mathbf{x} + \mathbf{y} \rVert \le \lVert \mathbf{x} \rVert + \lVert \mathbf{y} \rVert$
• Positive definite: $\lVert \mathbf{x} \rVert \ge 0$, and $\lVert \mathbf{x} \rVert = 0 \iff \mathbf{x} = \mathbf{0}$

The triangle inequality is just “a detour is never shorter than going direct.”

The common norms

For $\mathbf{x} \in \mathbb{R}^n$:

These are all special cases of the $p$-norm $\lVert \mathbf{x} \rVert_p = \left(\sum_i |x_i|^p\right)^{1/p}$.

The shape of a ruler: the unit ball

The clearest way to see the difference between norms is to draw the unit ball — every vector of length exactly 1. For the Euclidean norm that’s a circle; for Manhattan it’s a diamond; for the max norm it’s a square. Watch the unit ball morph as $p$ slides from 1 upward:

Same vectors, different rulers — and the shape of “length 1” is completely different. That shape is exactly why L1 and L2 behave so differently in ML.

NumPy

import numpy as np
 
x = np.array([3.0, 4.0])
 
print("L1  (Manhattan):", np.linalg.norm(x, 1))     # 7.0
print("L2  (Euclidean):", np.linalg.norm(x, 2))     # 5.0
print("Linf (max)     :", np.linalg.norm(x, np.inf))# 4.0
 
# L2 is the default
print("default norm   :", np.linalg.norm(x))        # 5.0

import numpy as np
 
x = np.array([3.0, 4.0])
 
print("L1  (Manhattan):", np.linalg.norm(x, 1))     # 7.0
print("L2  (Euclidean):", np.linalg.norm(x, 2))     # 5.0
print("Linf (max)     :", np.linalg.norm(x, np.inf))# 4.0
 
# L2 is the default
print("default norm   :", np.linalg.norm(x))        # 5.0

L1  (Manhattan): 7.0
L2  (Euclidean): 5.0
Linf (max)     : 4.0
default norm   : 5.0

L1  (Manhattan): 7.0
L2  (Euclidean): 5.0
Linf (max)     : 4.0
default norm   : 5.0

Why this matters for ML

• L2 (Ridge) regularization adds $\lVert \mathbf{w} \rVert_2^2$ to the loss — it shrinks all weights smoothly toward zero. Its round unit ball has no corners.
• L1 (Lasso) regularization adds $\lVert \mathbf{w} \rVert_1$ — its diamond unit ball has corners on the axes, which is precisely why Lasso drives some weights to exactly zero (automatic feature selection).
• Distance metrics in k-NN and clustering are norms of difference vectors; switching the norm changes which points count as “near.”

🧪 Try It Yourself

Exercise 1 – Compute three norms

Exercise 2 – Verify the triangle inequality

Exercise 3 – Why Lasso zeros weights

Recap

• A norm measures a vector’s length; it must be homogeneous, obey the triangle inequality, and be positive definite.
• The common norms are L1 (Manhattan), L2 (Euclidean), and L∞ (max) — different rulers with differently-shaped unit balls (diamond, circle, square).
• The norm you pick has real consequences: L1 → sparse (Lasso), L2 → smooth shrinkage (Ridge).

Next: the object that most norms come from — the inner product.