If linear independence tells you which vectors are not redundant, basis and rank tell you the right number of them. A basis is the smallest toolkit that still builds everything; rank counts how many independent directions a matrix actually has. Together they’re the backbone of PCA, compression, and “how many dimensions does my data really need?”

A real-life example: describing any location in a city

To pin down any point in a city you need exactly two directions — say “blocks East” and “blocks North.” One direction isn’t enough (you can only reach a line); a third (“blocks Northeast”) is redundant. Those two directions are a basis for the city plane, and the number 2 is its dimension. Swap to “blocks along Main St” and “blocks along 1st Ave” and you have a different basis for the same plane — the coordinates change, the place doesn’t.

Generating set, span, and basis

• The span of a set $\mathcal{A} = \{\mathbf{x}_1,\dots,\mathbf{x}_k\}$ is every vector you can build as a linear combination of them: $\text{span}[\mathcal{A}]$.
• If $\text{span}[\mathcal{A}] = V$, then $\mathcal{A}$ is a generating set of $V$.
• A basis is a minimal generating set — equivalently, a maximal linearly independent set. Remove any vector and it no longer spans; add any vector and it’s no longer independent.

Every vector in $V$ has a unique representation in a given basis. The canonical (standard) basis of $\mathbb{R}^3$ is

but there are infinitely many others — and all bases of a space have the same number of vectors. That number is the dimension, $\dim(V)$.

A basis is a coordinate system

Here’s the key mental model: choosing a basis is choosing a coordinate system. The same point $\mathbf{x}$ has different coordinates in different bases, but it’s the same point in the same plane. Watch a fixed black point keep its identity while an alternate (amber) basis rotates — its coordinates $(c_1, c_2)$ in that basis change continuously, yet $c_1\mathbf{b}_1 + c_2\mathbf{b}_2$ always lands on the same spot:

When the two amber vectors line up (collinear), the little warning fires: they no longer span the plane, so they’re not a basis — exactly the independence condition from the previous page.

Rank: how many independent directions?

The rank of a matrix $A$, written $\text{rk}(A)$, is the number of linearly independent columns — which (remarkably) always equals the number of linearly independent rows. Compute it by row-reducing and counting pivots.

  flowchart TD
  A["Matrix A (m × n)"] --> R["Row-reduce, count pivots"]
  R --> K["rk(A) = number of pivots"]
  K --> P1["rk(A) = n  →  columns independent, A invertible (if square)"]
  K --> P2["rk(A) = min(m, n)  →  FULL RANK"]
  K --> P3["rk(A) < min(m, n)  →  RANK DEFICIENT (redundancy)"]

Key properties that show up constantly:

• $\text{rk}(A) = \text{rk}(A^\top)$ — column rank = row rank.
• A square $A \in \mathbb{R}^{n\times n}$ is invertible $\iff \text{rk}(A) = n$.
• $A\mathbf{x} = \mathbf{b}$ is solvable $\iff \text{rk}(A) = \text{rk}([A \mid \mathbf{b}])$.
• The null space of $A$ has dimension $n - \text{rk}(A)$.
• Full rank means $\text{rk}(A) = \min(m, n)$; anything less is rank deficient.

Finding a basis of a subspace

To get a basis of a subspace spanned by some vectors:

1. Write the spanning vectors as columns of a matrix $A$.
2. Row-reduce $A$ to row-echelon form.
3. The original vectors sitting in pivot columns form a basis.

Same elimination, third job: earlier it solved systems and tested independence; now it extracts a basis.

NumPy: rank, dimension, and low-rank structure

import numpy as np
 
# Three vectors in R^3, but the third is the sum of the first two
A = np.column_stack([[1, 0, 1],
                     [0, 1, 1],
                     [1, 1, 2]]).astype(float)
 
rank = np.linalg.matrix_rank(A)
print("rank:", rank)                     # 2  -> only 2 independent directions
print("null space dimension:", A.shape[1] - rank)   # 3 - 2 = 1
 
# Real ML flavor: a "data matrix" that secretly lives in 1 dimension
x = np.linspace(0, 1, 50)
data = np.column_stack([x, 2 * x, -x])   # every column is a multiple of x
print("data shape:", data.shape, "-> rank:", np.linalg.matrix_rank(data))

import numpy as np
 
# Three vectors in R^3, but the third is the sum of the first two
A = np.column_stack([[1, 0, 1],
                     [0, 1, 1],
                     [1, 1, 2]]).astype(float)
 
rank = np.linalg.matrix_rank(A)
print("rank:", rank)                     # 2  -> only 2 independent directions
print("null space dimension:", A.shape[1] - rank)   # 3 - 2 = 1
 
# Real ML flavor: a "data matrix" that secretly lives in 1 dimension
x = np.linspace(0, 1, 50)
data = np.column_stack([x, 2 * x, -x])   # every column is a multiple of x
print("data shape:", data.shape, "-> rank:", np.linalg.matrix_rank(data))

rank: 2
null space dimension: 1
data shape: (50, 3) -> rank: 1

rank: 2
null space dimension: 1
data shape: (50, 3) -> rank: 1

That last line is the whole idea behind compression: a $50\times 3$ table that looks 3-dimensional is really rank 1 — one direction explains all of it.

Why this matters for ML

• PCA finds an orthonormal basis ordered by variance and keeps the top few vectors — a basis change plus a rank reduction.
• Low-rank approximation compresses images and recommendation matrices: store a rank-$k$ stand-in instead of the full grid.
• Effective dimensionality: matrix_rankmatrix_rank on your data tells you how many features are truly independent — often far fewer than the column count.

🧪 Try It Yourself

Exercise 1 – Compute the rank

Exercise 2 – Dimension of the null space

Exercise 3 – Detect hidden low-rank structure

Recap

• Span = all linear combinations; a generating set spans the whole space.
• A basis is a minimal generating set = a maximal independent set; every vector has a unique representation in it, and choosing one is choosing a coordinate system.
• All bases share the same size — the dimension of the space.
• Rank = number of independent columns (= rows) = number of pivots; it decides invertibility, solvability, and null-space size.
• Low rank means hidden redundancy — the mathematical basis of compression and PCA.

Next: functions that respect all this structure — every one of them secretly a matrix — Linear Mappings.