Some vectors add a genuinely new direction; others are just recycled combinations of the ones you already have. Telling these apart is linear independence — arguably the single most important concept in linear algebra, and the reason ML engineers worry about “redundant features” and “collinearity.”

A real-life example: giving directions in East Africa

You’re in Nairobi and want to describe where Kigali is. You say:

“Go 506 km Northwest to Kampala, then 374 km Southwest.”

That’s enough — two directions pin down the location exactly. Now you add:

“It’s also about 751 km West of here.”

True, but redundant: the “751 km West” direction is already a combination of the first two. The Northwest and Southwest vectors are linearly independent (neither is a stretch of the other), but throwing in the West vector makes the set linearly dependent — it carries no new information.

That’s the whole intuition. Independent = no redundancy. Dependent = at least one vector is a combination of the others.

Linear combinations

Given vectors $\mathbf{x}_1, \dots, \mathbf{x}_k$ and scalars $\lambda_1, \dots, \lambda_k$, a linear combination is any weighted sum:

Note that $\mathbf{0}$ is always a linear combination — just take every $\lambda_i = 0$. The interesting question is whether there’s a non-trivial way (not all zeros) to combine the vectors and still land on $\mathbf{0}$.

The definition

Vectors $\mathbf{x}_1, \dots, \mathbf{x}_k$ are linearly dependent if there exist scalars $\lambda_1,\dots,\lambda_k$ — not all zero — with

If the only way to reach $\mathbf{0}$ is the trivial $\lambda_1 = \cdots = \lambda_k = 0$, the vectors are linearly independent.

  flowchart TD
  Q["Can Σ λᵢ xᵢ = 0
with some λᵢ ≠ 0 ?"]
  Q -->|"yes, a non-trivial combo exists"| D["Linearly DEPENDENT
(one vector is redundant)"]
  Q -->|"no, only all-zeros works"| I["Linearly INDEPENDENT
(every vector adds a direction)"]

See dependence happen

Two vectors in the plane are independent exactly when they don’t lie on the same line. Geometrically, independent vectors span a parallelogram with real area; as one vector rotates toward the other, that area shrinks to zero — and at the instant they’re collinear, the set becomes dependent. The number tracking this is the determinant (the signed area):

For two vectors, “independent” and “nonzero determinant” are the same statement. In higher dimensions the parallelogram becomes a parallelepiped, but the rule is identical: zero volume ⇔ dependent.

Handy shortcuts

Before doing any real work, these quick checks often settle it:

• Any set that contains $\mathbf{0}$ is automatically dependent.
• If two vectors are scalar multiples of each other, they’re dependent.
• In $\mathbb{R}^n$, more than $n$ vectors are always dependent (you can’t have more independent directions than the space has dimensions).

The reliable method: Gaussian elimination

For anything real, write the vectors as columns of a matrix and row-reduce. Then:

• Pivot columns mark the linearly independent vectors.
• Non-pivot columns are linear combinations of the pivots on their left.
• The vectors are all independent $\iff$ every column is a pivot column.

This is the same elimination from the solving-systems page — reused as an independence detector.

NumPy: rank tells you instantly

The rank of the matrix (number of pivots) equals the number of independent columns. Compare it to the number of vectors:

import numpy as np
 
def independent(*vectors):
    A = np.column_stack(vectors)
    return np.linalg.matrix_rank(A) == A.shape[1]
 
# The East-Africa example (2-D): NW and SW directions
nw = np.array([-3.0,  4.0])   # "Northwest"-ish
sw = np.array([-3.0, -2.0])   # "Southwest"-ish
print("NW, SW independent?", independent(nw, sw))          # True
 
# Add the redundant "West" vector = a combination of the other two
west = nw + sw                # literally a linear combination
print("NW, SW, West independent?", independent(nw, sw, west))   # False
 
# Two collinear vectors
a = np.array([2.0, 1.0])
b = np.array([4.0, 2.0])      # b = 2a
print("a, b independent?", independent(a, b))              # False

import numpy as np
 
def independent(*vectors):
    A = np.column_stack(vectors)
    return np.linalg.matrix_rank(A) == A.shape[1]
 
# The East-Africa example (2-D): NW and SW directions
nw = np.array([-3.0,  4.0])   # "Northwest"-ish
sw = np.array([-3.0, -2.0])   # "Southwest"-ish
print("NW, SW independent?", independent(nw, sw))          # True
 
# Add the redundant "West" vector = a combination of the other two
west = nw + sw                # literally a linear combination
print("NW, SW, West independent?", independent(nw, sw, west))   # False
 
# Two collinear vectors
a = np.array([2.0, 1.0])
b = np.array([4.0, 2.0])      # b = 2a
print("a, b independent?", independent(a, b))              # False

NW, SW independent? True
NW, SW, West independent? False
a, b independent? False

NW, SW independent? True
NW, SW, West independent? False
a, b independent? False

Why this matters for ML

• Multicollinearity: if two features are (near-)linearly dependent, the regression matrix $X^\top X$ becomes singular or ill-conditioned — coefficients blow up and become uninterpretable. Detecting dependence is detecting this bug.
• Feature selection / PCA: the goal is to keep an independent set of directions and drop the redundant ones.
• Rank of a data matrix tells you the true number of independent directions your data actually explores — often far fewer than the number of columns.

🧪 Try It Yourself

Exercise 1 – Rank-based independence test

Exercise 2 – Determinant as the 2-D area

Exercise 3 – Too many vectors

Recap

• A linear combination is a weighted sum $\sum_i \lambda_i \mathbf{x}_i$.
• Vectors are dependent if some non-trivial combination equals $\mathbf{0}$; otherwise independent.
• In 2-D, independent $\iff$ nonzero determinant (nonzero area); the volume test generalizes.
• The practical test is Gaussian elimination → pivot columns, or compare rank to the number of vectors.
• In ML this is exactly multicollinearity detection and the basis of feature reduction.

Next: the smallest independent set that still builds the whole space — Basis and Rank.