Systems of Linear Equations

Linear algebra was born from one deceptively simple question: given several linear constraints, what values satisfy all of them at once? That question is a system of linear equations, and it shows up everywhere — from mixing ingredients in a factory to fitting a regression line through data.

A real-life example: the production plan

Imagine a small workshop that makes two products, tables and chairs. Each needs wood and labor:

A table uses 4 units of wood and 2 hours of labor.
A chair uses 1 unit of wood and 2 hours of labor.

This week you have 5 units of wood and 6 hours of labor, and you want to use them up exactly (no waste). How many tables $x_1$ and chairs $x_2$ should you make?

Each resource gives you one equation:

\begin{aligned} 4x_1 + 1x_2 &= 5 \quad \text{(wood)}\\ 2x_1 + 2x_2 &= 6 \quad \text{(labor)} \end{aligned}

Two equations, two unknowns. This is a system of linear equations. Solving it (we will, below) tells you the unique plan that spends every resource.

The general form

With $m$ equations and $n$ unknowns $x_1, \dots, x_n$ , a system of linear equations looks like this:

\begin{aligned} a_{11}x_1 + a_{12}x_2 + \cdots + a_{1n}x_n &= b_1\\ a_{21}x_1 + a_{22}x_2 + \cdots + a_{2n}x_n &= b_2\\ &\;\;\vdots\\ a_{m1}x_1 + a_{m2}x_2 + \cdots + a_{mn}x_n &= b_m \end{aligned}

The $a_{ij}$ are known coefficients, the $b_i$ are known constants, and the $x_j$ are the unknowns we solve for. Any tuple $(x_1, \dots, x_n)$ that satisfies every equation simultaneously is called a solution.

The key question: how many solutions?

For a real-valued system, there are only three possibilities:

diagram Diagram mermaid

  flowchart TD
  S["System of linear
equations"]
  S --> A["Exactly ONE solution
(lines cross at a point)"]
  S --> B["NO solution
(parallel, never meet)"]
  S --> C["INFINITELY many
(same line, fully overlap)"]

The geometry makes this vivid. With two unknowns, each equation is a line in the $x_1x_2$ -plane. A solution must lie on all lines at once — so the solution set is the intersection of the lines:

Lines cross at one point → one solution.
Lines are parallel but distinct → no solution.
Lines are identical → infinitely many solutions.

See it move

Watch two lines interact. One line (amber) stays fixed; the other (blue) sweeps through all three regimes. The white dot marks the solution when it exists — notice how it flies off to infinity as the lines become parallel, then the lines merge into one:

sketch The three solution regimes of a 2-variable system p5.js

Two lines on the plane. The amber line is fixed; the blue line sweeps from crossing it (one solution) to parallel (no solution) to coincident (infinitely many). The white dot is the unique solution when it exists.

Right away this tells you something profound: most of the difficulty in solving a system is just figuring out which of these three cases you’re in. In machine learning, “no exact solution” is the normal case (more data points than parameters), which is precisely why we switch to least squares — finding the closest thing to a solution.

Solving our workshop system

Back to the tables and chairs:

\begin{aligned} 4x_1 + x_2 &= 5\\ 2x_1 + 2x_2 &= 6 \end{aligned}

From the second equation, $x_1 + x_2 = 3$ , so $x_2 = 3 - x_1$ . Substitute into the first:

4x_1 + (3 - x_1) = 5 \;\Longrightarrow\; 3x_1 = 2 \;\Longrightarrow\; x_1 = \tfrac{2}{3}.

Then $x_2 = 3 - \tfrac{2}{3} = \tfrac{7}{3}$ . A single, unique answer — the two lines cross at exactly one point.

Let NumPy do it

For anything bigger than 2×2 you’ll reach for code. NumPy’s linalg.solvelinalg.solve finds the unique solution directly:

solve_system.py

import numpy as np
 
# 4*x1 + 1*x2 = 5   (wood)
# 2*x1 + 2*x2 = 6   (labor)
A = np.array([[4.0, 1.0],
              [2.0, 2.0]])
b = np.array([5.0, 6.0])
 
x = np.linalg.solve(A, b)
print("tables x1 =", round(x[0], 4))
print("chairs x2 =", round(x[1], 4))
 
# Verify: plug the solution back in
print("check A @ x =", A @ x)   # should equal b

solve_system.py

import numpy as np
 
# 4*x1 + 1*x2 = 5   (wood)
# 2*x1 + 2*x2 = 6   (labor)
A = np.array([[4.0, 1.0],
              [2.0, 2.0]])
b = np.array([5.0, 6.0])
 
x = np.linalg.solve(A, b)
print("tables x1 =", round(x[0], 4))
print("chairs x2 =", round(x[1], 4))
 
# Verify: plug the solution back in
print("check A @ x =", A @ x)   # should equal b

text

tables x1 = 0.6667
chairs x2 = 2.3333
check A @ x = [5. 6.]

text

tables x1 = 0.6667
chairs x2 = 2.3333
check A @ x = [5. 6.]

The three regimes, side by side

diagram Diagram mermaid

  flowchart LR
  subgraph One["Unique"]
    direction TB
    O1["4x₁ + x₂ = 5
2x₁ + 2x₂ = 6"] --> O2["cross once"]
  end
  subgraph None["No solution"]
    direction TB
    N1["x₁ + x₂ = 3
x₁ + x₂ = 1"] --> N2["parallel"]
  end
  subgraph Inf["Infinite"]
    direction TB
    I1["x₁ + x₂ = 3
2x₁ + 2x₂ = 6"] --> I2["same line"]
  end

Why this matters for ML

Linear regression solves a system like this to fit a line/plane to data — except there are usually more equations than unknowns, so no exact solution exists and we find the best-fit instead (Chapter 9).
The “singular matrix” error you’ll hit in practice is this exact theory surfacing: redundant or contradictory features make the system unsolvable.
The compact form $A\mathbf{x} = \mathbf{b}$ we’re heading toward is the single most common equation in all of numerical computing.

🧪 Try It Yourself

Exercise 1 – Solve a 2×2 system

Exercise 2 – Detect an unsolvable system

Exercise 3 – Verify a solution

Recap

A system of linear equations asks for values satisfying several linear constraints at once.
Geometrically each equation is a line (or plane); solutions are their intersection.
There are exactly three outcomes: one, none, or infinitely many solutions.
np.linalg.solvenp.linalg.solve handles the unique case; a Singular matrixSingular matrix error signals the other two.

Next: we compress this whole system into a single object — the matrix — and learn how it both stores and transforms data.

If this helped you, consider buying me a coffee ☕

Buy me a coffee