Probability starts with a careful setup: a probability space built from three ingredients (a sample space, an event space, and a probability measure), plus a random variable that maps messy real-world outcomes to numbers we can compute with. Getting this foundation straight untangles the notation that trips up most beginners in probabilistic machine learning.

A real-life example: drawing coins from a bag

A bag holds US coins ($) and UK coins (£). You draw two, with replacement; a draw returns a$ with probability 0.3. You care about **how many $** you got — 0, 1, or 2. That count is a **random variable**: a function turning the raw outcome (like "£ then$”) into a number (1). The whole machinery below just makes this everyday move precise.

The three ingredients

A probability space $(\Omega, \mathcal{A}, P)$ models a random experiment:

• Sample space $\Omega$ — the set of all possible outcomes. Two coin draws: \Omega = \{\$, $£, £$, ££}$.
• Event space $\mathcal{A}$ — the collection of events (subsets of $\Omega$) we can ask about. For discrete problems it’s usually all subsets.
• Probability measure $P$ — assigns each event $A$ a number $P(A) \in [0, 1]$, with the whole space summing to one: $P(\Omega) = 1$.

The random variable

We rarely work with $\Omega$ directly. Instead a random variable $X : \Omega \to \mathcal{T}$ maps each outcome to a value in a target space $\mathcal{T}$ (often numbers). For “count the $s”:

The distribution (or law) of $X$ assigns probabilities to its values. With P(\) = 0.3$ and independent draws:

Probability = long-run frequency

One way to read $P(X{=}k)$: repeat the experiment many times, and the fraction of times you see value $k$ settles toward $P(X{=}k)$. Watch the bars (empirical frequencies) converge to the true distribution (amber ticks) as draws accumulate — the law of large numbers in action:

The empirical bars wobble at first, then lock onto the true probabilities — this is why we can estimate distributions from data.

Probability vs. statistics

• Probability: given a model, predict what data looks like (forward).
• Statistics: given data, infer the model that produced it (backward).

Machine learning is mostly the second — building a model that explains observed data — which is why probability is its foundation.

  flowchart LR
  O["Sample space Ω
(all outcomes)"] --> X["Random variable X : Ω → T"]
  X --> D["Distribution P_X
(probabilities over values)"]
  D -.-> EST["estimate from data (statistics / ML)"]

NumPy

import numpy as np
 
rng = np.random.default_rng(0)
 
# Random variable X = number of $ in two draws, P($) = 0.3
def draw_X():
    return int(rng.random() < 0.3) + int(rng.random() < 0.3)
 
# Simulate and estimate the distribution (law of large numbers)
N = 200_000
samples = np.array([draw_X() for _ in range(N)])
for k in [0, 1, 2]:
    print(f"P(X={k}) empirical = {np.mean(samples == k):.3f}")
 
print("true:", {0: 0.49, 1: 0.42, 2: 0.09})

import numpy as np
 
rng = np.random.default_rng(0)
 
# Random variable X = number of $ in two draws, P($) = 0.3
def draw_X():
    return int(rng.random() < 0.3) + int(rng.random() < 0.3)
 
# Simulate and estimate the distribution (law of large numbers)
N = 200_000
samples = np.array([draw_X() for _ in range(N)])
for k in [0, 1, 2]:
    print(f"P(X={k}) empirical = {np.mean(samples == k):.3f}")
 
print("true:", {0: 0.49, 1: 0.42, 2: 0.09})

P(X=0) empirical = 0.490
P(X=1) empirical = 0.420
P(X=2) empirical = 0.090
true: {0: 0.49, 1: 0.42, 2: 0.09}

P(X=0) empirical = 0.490
P(X=1) empirical = 0.420
P(X=2) empirical = 0.090
true: {0: 0.49, 1: 0.42, 2: 0.09}

Why this matters for ML

• Everything is a random variable: features, labels, weights, and predictions are all modeled as random variables with distributions.
• The frequentist view (probability = long-run frequency) justifies estimating probabilities by counting — the basis of empirical risk and validation.
• Clean separation of $\Omega$, $\mathcal{T}$, and $X$ demystifies the lazy notation $p(x)$ you’ll see everywhere.

🧪 Try It Yourself

Exercise 1 – A random variable as a lookup

Exercise 2 – Distribution sums to one

Exercise 3 – Estimate a probability by simulation

Recap

• A probability space $(\Omega, \mathcal{A}, P)$ has a sample space, event space, and probability measure ($P(\Omega) = 1$).
• A random variable $X : \Omega \to \mathcal{T}$ maps outcomes to a target space; its distribution $P_X$ assigns probabilities to values.
• Probabilities are long-run frequencies — the reason we can estimate them from data.

Next: the two flavors of distribution — discrete and continuous.