Probability comes in two flavors depending on whether outcomes are countable (discrete) or form a continuum (continuous). Discrete variables use a probability mass function; continuous ones use a probability density function and a cumulative distribution function. Getting the difference straight — especially that a density can exceed 1 — is essential for reading any probabilistic ML model.

A real-life example: category vs. measurement

A user’s country is discrete (a finite set of labels) — model it with a probability mass function that assigns a probability to each country. A user’s height is continuous (any real number in a range) — model it with a probability density function, where only intervals (“between 170 and 180 cm”) have nonzero probability. Categorical → mass; measured → density.

Discrete: the probability mass function

For a discrete random variable, the probability mass function (pmf) gives the probability of each value directly:

With two variables, the joint pmf $p(x, y) = P(X{=}x, Y{=}y)$ fills a table. Summing across a row or column gives the marginal; restricting to one row/column and renormalizing gives the conditional:

Continuous: density and cumulative functions

A probability density function (pdf) $f(x)$ satisfies two rules:

Probability comes from integrating the density over an interval: $P(a \le X \le b) = \int_a^b f(x)\,dx$. The cumulative distribution function (cdf) accumulates this from the left:

Watch the pdf integrate into the cdf

The top panel is a pdf $f(x)$; the shaded area to the left of the sweeping line is $P(X \le x)$. The bottom panel plots that accumulated area — the cdf — climbing from 0 to 1 as the line sweeps right:

The cdf is just “how much probability have we accumulated so far” — it always starts at 0, ends at 1, and never decreases.

Discrete vs. continuous, side by side

  flowchart TD
  RV["Random variable"] --> DISC["Discrete (countable)"]
  RV --> CONT["Continuous (interval)"]
  DISC --> PMF["pmf: P(X = x)
probabilities sum to 1"]
  CONT --> PDF["pdf: f(x) ≥ 0
integrates to 1 (can exceed 1)"]
  PMF --> J1["point probability P(X = x)"]
  PDF --> J2["interval probability ∫ f(x) dx"]
  PDF --> CDF["cdf: F(x) = P(X ≤ x)"]

NumPy

import numpy as np
 
# Discrete: pmf of a fair die — each face 1/6, sums to 1
pmf = np.full(6, 1/6)
print("pmf sums to 1:", np.isclose(pmf.sum(), 1.0))
 
# Continuous uniform on [0.9, 1.6]: density height > 1
a, b = 0.9, 1.6
height = 1 / (b - a)
print(f"uniform density height = {height:.3f} (> 1, but area = {(b-a)*height:.1f})")
 
# pdf -> cdf by cumulative integration (trapezoid) for a standard normal
x = np.linspace(-4, 4, 2001)
pdf = np.exp(-0.5 * x**2) / np.sqrt(2 * np.pi)
cdf = np.cumsum(pdf) * (x[1] - x[0])
print("cdf at x=0  ≈", round(cdf[np.argmin(abs(x))], 3))   # ~0.5
print("cdf at x=4  ≈", round(cdf[-1], 3))                   # ~1.0

import numpy as np
 
# Discrete: pmf of a fair die — each face 1/6, sums to 1
pmf = np.full(6, 1/6)
print("pmf sums to 1:", np.isclose(pmf.sum(), 1.0))
 
# Continuous uniform on [0.9, 1.6]: density height > 1
a, b = 0.9, 1.6
height = 1 / (b - a)
print(f"uniform density height = {height:.3f} (> 1, but area = {(b-a)*height:.1f})")
 
# pdf -> cdf by cumulative integration (trapezoid) for a standard normal
x = np.linspace(-4, 4, 2001)
pdf = np.exp(-0.5 * x**2) / np.sqrt(2 * np.pi)
cdf = np.cumsum(pdf) * (x[1] - x[0])
print("cdf at x=0  ≈", round(cdf[np.argmin(abs(x))], 3))   # ~0.5
print("cdf at x=4  ≈", round(cdf[-1], 3))                   # ~1.0

pmf sums to 1: True
uniform density height = 1.429 (> 1, but area = 1.0)
cdf at x=0  ≈ 0.5
cdf at x=4  ≈ 1.0

pmf sums to 1: True
uniform density height = 1.429 (> 1, but area = 1.0)
cdf at x=0  ≈ 0.5
cdf at x=4  ≈ 1.0

Why this matters for ML

• Discrete distributions model categorical features and class labels (softmax outputs are a pmf over classes).
• Continuous densities model measurements, weights, and noise; likelihoods are products of densities.
• Joint / marginal / conditional structure is the vocabulary of graphical models, and the reason “marginalize out the nuisance variable” is a core ML operation.

🧪 Try It Yourself

Exercise 1 – A valid pmf

Exercise 2 – Marginal from a joint table

Exercise 3 – Density can exceed 1

Recap

• Discrete variables use a pmf ($P(X{=}x)$, sums to 1); continuous ones use a pdf ($f(x) \ge 0$, integrates to 1) and a cdf ($F(x) = P(X \le x)$).
• A density is not a probability — it can exceed 1; only the area (an interval’s integral) is a probability.
• Joint → marginal (sum/integrate out) → conditional (restrict and renormalize) is the core toolkit.

Next: the two rules that generate all of probability — sum, product, and Bayes.