The inner product is the master key of this chapter. From it we get lengths, distances, angles, and orthogonality — every geometric notion follows once you can compute $\langle \mathbf{x}, \mathbf{y}\rangle$. You already know one inner product (the dot product); here we see the general version and why it matters for ML.

A real-life example: weighted similarity

You compare two users by their ratings of five movies. A plain dot product treats all movies equally. But maybe agreement on a niche art film is more informative than agreement on a blockbuster everyone likes. An inner product lets you weight each dimension differently — $\langle \mathbf{x}, \mathbf{y}\rangle = \mathbf{x}^\top A\,\mathbf{y}$ with a weight matrix $A$ — so “similarity” reflects what actually matters. That’s an inner product that is not the dot product.

The dot product you know

The familiar dot product on $\mathbb{R}^n$ is

It’s the special case everyone starts with. The general concept keeps its useful properties but allows other definitions.

General inner products

An inner product is a bilinear mapping $\langle \cdot, \cdot\rangle : V \times V \to \mathbb{R}$ that is symmetric and positive definite:

• Bilinear: linear in each argument separately.
• Symmetric: $\langle \mathbf{x}, \mathbf{y}\rangle = \langle \mathbf{y}, \mathbf{x}\rangle$.
• Positive definite: $\langle \mathbf{x}, \mathbf{x}\rangle > 0$ for $\mathbf{x}\neq\mathbf{0}$, and $\langle \mathbf{0}, \mathbf{0}\rangle = 0$.

A vector space with an inner product is an inner product space; with the dot product it’s a Euclidean vector space.

  flowchart TD
  BL["Bilinear map Ω(x, y)"] --> S["+ symmetric"]
  S --> PD["+ positive definite"]
  PD --> IP["= Inner product ⟨x, y⟩"]
  IP --> USE["lengths, angles, orthogonality"]

Not every inner product is the dot product

On $\mathbb{R}^2$, this is a valid inner product but different from the dot product:

The geometric meaning of the dot product

For the standard dot product, $\langle \mathbf{x}, \mathbf{y}\rangle = \lVert \mathbf{x}\rVert\, \lVert \mathbf{y}\rVert\cos\omega$. So the sign of the inner product tells you the relationship between two vectors: positive when they point similar ways, zero when perpendicular, negative when opposed. Rotate the blue vector and watch the value — and its sign — change:

Symmetric positive definite matrices

Every inner product on an $n$-dimensional space can be written, in coordinates, as

where $A$ is a symmetric positive definite (SPD) matrix ($A = A^\top$ and $\mathbf{x}^\top A \mathbf{x} > 0$ for all $\mathbf{x}\neq\mathbf{0}$). The dot product is the case $A = I$. SPD matrices are everywhere in ML: covariance matrices, kernel/Gram matrices, and the curvature (Hessians) of convex losses are all SPD.

NumPy

import numpy as np
 
x = np.array([1.0, 2.0])
y = np.array([3.0, -1.0])
 
# Standard dot product
print("dot product:", x @ y)                     # 1
 
# A custom inner product via an SPD matrix A
A = np.array([[2.0, 0.0],
              [0.0, 1.0]])
print("SPD is symmetric:", np.allclose(A, A.T))
print("weighted inner product:", x @ A @ y)      # 2*1*3 + 1*2*(-1) = 4
 
# Check A is positive definite: all eigenvalues > 0
print("eigenvalues:", np.linalg.eigvalsh(A), "-> PD:", np.all(np.linalg.eigvalsh(A) > 0))

import numpy as np
 
x = np.array([1.0, 2.0])
y = np.array([3.0, -1.0])
 
# Standard dot product
print("dot product:", x @ y)                     # 1
 
# A custom inner product via an SPD matrix A
A = np.array([[2.0, 0.0],
              [0.0, 1.0]])
print("SPD is symmetric:", np.allclose(A, A.T))
print("weighted inner product:", x @ A @ y)      # 2*1*3 + 1*2*(-1) = 4
 
# Check A is positive definite: all eigenvalues > 0
print("eigenvalues:", np.linalg.eigvalsh(A), "-> PD:", np.all(np.linalg.eigvalsh(A) > 0))

dot product: 1.0
SPD is symmetric: True
weighted inner product: 4.0
eigenvalues: [1. 2.] -> PD: True

dot product: 1.0
SPD is symmetric: True
weighted inner product: 4.0
eigenvalues: [1. 2.] -> PD: True

Why this matters for ML

• Cosine similarity — the workhorse for comparing embeddings — is an inner product divided by the norms.
• Kernels (SVMs, Gaussian processes) are inner products in a high-dimensional feature space, computed cheaply; the kernel matrix is SPD.
• Covariance matrices are SPD and define a natural (Mahalanobis) inner product that accounts for feature scale and correlation.

🧪 Try It Yourself

Exercise 1 – Dot product and its sign

Exercise 2 – A weighted (SPD) inner product

Exercise 3 – Is the matrix positive definite?

Recap

• An inner product is a symmetric, positive-definite bilinear map; the dot product is one example.
• Its sign encodes geometry: + similar, 0 perpendicular, − opposed.
• Every inner product is $\hat{\mathbf{x}}^\top A\,\hat{\mathbf{y}}$ for a symmetric positive definite matrix $A$ — the same objects as covariance and kernel matrices in ML.

Next: use the inner product to measure lengths and distances.