The inner product doesn’t just measure length — it measures angle. That gives us cosine similarity (how ML compares embeddings) and orthogonality (the “right angle” that underlies PCA, whitening, and rotations). This page turns the inner product into an angle-meter.

A real-life example: document similarity

Represent two documents as vectors of word counts. Their angle tells you how similar they are regardless of length: a short tweet and a long article about the same topic point in nearly the same direction (small angle) even though one vector is much longer. That angle — via its cosine — is the standard “cosine similarity” behind search and recommendation.

The angle between two vectors

For nonzero $\mathbf{x}, \mathbf{y}$, Cauchy-Schwarz guarantees

so there’s a unique angle $\omega \in [0, \pi]$ with

The quantity on the right is cosine similarity: $+1$ means same direction, $0$ means perpendicular, $-1$ means opposite.

Watch the angle

Rotate the blue vector and watch $\omega$ and its cosine. The angle is largest when the vectors oppose, zero when aligned, and the vectors turn orthogonal the instant $\cos\omega = 0$:

Orthogonality

Two vectors are orthogonal — written $\mathbf{x} \perp \mathbf{y}$ — exactly when

If they are also unit length ($\lVert\mathbf{x}\rVert = \lVert\mathbf{y}\rVert = 1$), they are orthonormal. Orthogonality is the generalization of “perpendicular” to any inner product — and the $\mathbf{0}$ vector is orthogonal to everything.

Orthogonal matrices

A square matrix $A$ is orthogonal if its columns are orthonormal, which is equivalent to

The inverse is just the transpose — no elimination needed. Orthogonal matrices are special because they preserve lengths and angles: $\lVert A\mathbf{x}\rVert = \lVert \mathbf{x}\rVert$ and the angle between $A\mathbf{x}$ and $A\mathbf{y}$ equals the angle between $\mathbf{x}$ and $\mathbf{y}$. Geometrically they are rotations (possibly with a flip) — the subject of a later page.

  flowchart TD
  IP["⟨x, y⟩"] --> COS["cos ω = ⟨x,y⟩ / (‖x‖‖y‖)"]
  COS --> ORTH["⟨x, y⟩ = 0 → orthogonal"]
  ORTH --> ONM["orthonormal (also unit length)"]
  ONM --> OM["orthogonal matrix: AᵀA = I, A⁻¹ = Aᵀ"]
  OM --> PRES["preserves lengths & angles → rotation"]

NumPy

import numpy as np
 
def angle_deg(x, y):
    cos = (x @ y) / (np.linalg.norm(x) * np.linalg.norm(y))
    return np.degrees(np.arccos(np.clip(cos, -1, 1)))
 
x = np.array([1.0, 1.0])
y = np.array([1.0, 2.0])
print("angle:", round(angle_deg(x, y), 2), "degrees")   # ~18.43
 
# Orthogonal pair
a = np.array([1.0, 0.0])
b = np.array([0.0, 1.0])
print("⟨a, b⟩ =", a @ b, "-> orthogonal:", np.isclose(a @ b, 0))
 
# An orthogonal matrix (45° rotation): Q.T @ Q = I
t = np.pi / 4
Q = np.array([[np.cos(t), -np.sin(t)],
              [np.sin(t),  np.cos(t)]])
print("QᵀQ = I:", np.allclose(Q.T @ Q, np.eye(2)))
print("length preserved:", np.allclose(np.linalg.norm(Q @ x), np.linalg.norm(x)))

import numpy as np
 
def angle_deg(x, y):
    cos = (x @ y) / (np.linalg.norm(x) * np.linalg.norm(y))
    return np.degrees(np.arccos(np.clip(cos, -1, 1)))
 
x = np.array([1.0, 1.0])
y = np.array([1.0, 2.0])
print("angle:", round(angle_deg(x, y), 2), "degrees")   # ~18.43
 
# Orthogonal pair
a = np.array([1.0, 0.0])
b = np.array([0.0, 1.0])
print("⟨a, b⟩ =", a @ b, "-> orthogonal:", np.isclose(a @ b, 0))
 
# An orthogonal matrix (45° rotation): Q.T @ Q = I
t = np.pi / 4
Q = np.array([[np.cos(t), -np.sin(t)],
              [np.sin(t),  np.cos(t)]])
print("QᵀQ = I:", np.allclose(Q.T @ Q, np.eye(2)))
print("length preserved:", np.allclose(np.linalg.norm(Q @ x), np.linalg.norm(x)))

angle: 18.43 degrees
⟨a, b⟩ = 0.0 -> orthogonal: True
QᵀQ = I: True
length preserved: True

angle: 18.43 degrees
⟨a, b⟩ = 0.0 -> orthogonal: True
QᵀQ = I: True
length preserved: True

Why this matters for ML

• Cosine similarity is the default metric for comparing text/image embeddings — it ignores magnitude and compares direction.
• Orthogonal features are uncorrelated, which stabilizes training and interpretation (whitening decorrelates them).
• Orthogonal matrices are used in stable numerical routines (QR, SVD) and as weight constraints that prevent exploding/vanishing signals in deep nets.

🧪 Try It Yourself

Exercise 1 – Cosine similarity

Exercise 2 – Test orthogonality

Exercise 3 – Orthogonal matrices preserve length

Recap

• The inner product gives the angle: $\cos\omega = \langle\mathbf{x},\mathbf{y}\rangle / (\lVert\mathbf{x}\rVert\lVert\mathbf{y}\rVert)$ — this is cosine similarity.
• Orthogonal means $\langle\mathbf{x},\mathbf{y}\rangle = 0$; orthonormal adds unit length.
• Orthogonal matrices ($A^{-1} = A^\top$) preserve lengths and angles — they’re rotations.

Next: a basis made entirely of orthonormal vectors — the orthonormal basis.