Data, Models, and Learning

Part I built the mathematics. This chapter is the vocabulary you need before any of it can be pointed at a dataset — and the book is unusually careful here, because three of the words involved mean more than one thing.

The chapter’s own framing: there are three components of a machine learning system — data, models and learning — and the guiding question is “what do we mean by good models?” The answer the book commits to is that good models perform well on unseen data, which sounds obvious and turns out to drive everything in §8.2 onward.

What you’ll learn

• Why “data as vectors” is a sequence of modelling decisions, not a formatting step — reconstructed from the book’s Tables 8.1 and 8.2.
• Measured: latitude and salary in Table 8.2 differ in spread by a factor of 297,010, which is why the book’s advice to standardise every column exists.
• Equation 8.2, a model as a function, and the trick of Equations 8.4–8.5 that turns an affine model into a linear one.
• Section 8.1.3, a model as a distribution, and precisely what extra question that lets you ask.
• The book’s own running example fitted: $\theta = (8.807355,\ 2.122453)$, $f(60) = 136.154562$ with a residual scale of $25.603199$.
• Section 8.1.4’s three algorithmic phases, and the naming collision the book explicitly warns about.
• Which chapter answers which phase, so the rest of Part II has a map.

Intuition: three problems wearing one word

“Data” sounds like something you are given. It is not. Between the spreadsheet a domain expert hands you and the matrix $\mathbf{X} \in \mathbb{R}^{N\times D}$ that an algorithm consumes, somebody makes half a dozen irreversible choices. Table 8.1 to Table 8.2 in the book is exactly that transition, and this page counts the choices.

“Model” means two different things, and the book is explicit that it will “revisit it multiple times”. A model can be a function — feed it features, get a number — or a distribution, which describes how likely each answer is. Both appear in this chapter; the first drives §8.2, the second §8.3 and §8.4.

“Learning” means three different things, depending on which phase you are in. Predicting is not training and training is not model selection, and they consume different slices of your data.

diagram Section 8.1, and where each thread goes mermaid

  flowchart TD
  RAW["a table of strings, categories,
locations, numbers"]
  RAW -->|"four decisions"| VEC["X in R^(N x D), y in R^N
Section 8.1.1"]
  VEC --> FN["model as a FUNCTION
Eq 8.2, f(x) = theta-transpose x + theta-0"]
  VEC --> PR["model as a DISTRIBUTION
Section 8.1.3"]
  FN --> ERM["empirical risk minimization
Section 8.2"]
  PR --> PE["parameter estimation
Section 8.3"]
  PR --> BI["Bayesian inference
Section 8.4"]
  ERM --> Q["what do we mean by GOOD?
performs well on UNSEEN data"]
  PE --> Q
  BI --> Q
  Q --> MS["model selection
Section 8.6"]

§8.1.1 Data as vectors

The book’s assumption is that data is tabular and tidy (Wickham, 2014; Codd, 1990): each row is an example, each column is a feature. It also states plainly what it is not going to cover — identifying good features, which “depend on domain expertise and require careful engineering”.

But even given a tidy table, getting to numbers takes decisions. Here is Table 8.1 as the book prints it:

Name	Gender	Degree	Postcode	Age	Annual salary
Aditya	M	MSc	W21BG	36	89563
Bob	M	PhD	EC1A1BA	47	123543
Chloé	F	BEcon	SW1A1BH	26	23989
Daisuke	M	BSc	SE207AT	68	138769
Elisabeth	F	MBA	SE10AA	33	113888

and here is Table 8.2, the numerical version:

Gender ID	Degree	Latitude	Longitude	Age	Annual Salary (thousands)
$-1$	2	51.5073	0.1290	36	89.563
$-1$	3	51.5074	0.1275	47	123.543
$+1$	1	51.5071	0.1278	26	23.989
$-1$	1	51.5075	0.1281	68	138.769
$+1$	2	51.5074	0.1278	33	113.888

Four decisions happened. Each is defensible and each is a claim about the world:

1. The Name column was dropped. The book gives two reasons: it is not expected to be informative, and dropping it anonymises the row. Note these are different kinds of reason — one is about prediction, the other about ethics, and they happen to agree here.
2. Gender became $-1$ / $+1$. The book explicitly notes $0$ / $1$ would also do. That choice is invisible to an unregularised linear model and not invisible to a regularised one, since it changes what “a large coefficient” means.
3. Degree became $1$ / $2$ / $3$. This asserts an order — bachelor’s, master’s, PhD — and simultaneously asserts that BEcon $=$ BSc and MBA $=$ MSc. Both are domain claims, and the encoding also asserts that the step from $1$ to $2$ equals the step from $2$ to $3$.
4. Postcode became a latitude and longitude. This one needs actual knowledge: that “SE10AA” is not a string but a place in London.

The advice about scaling is not cosmetic

The book says: “Without additional information, one should shift and scale all columns of the dataset such that they have an empirical mean of 0 and an empirical variance of 1.”

Measured on Table 8.2: the latitude column has standard deviation $0.000136$ and the salary column $40.288419$ — a ratio of 297,010. A penalty term $\lVert\boldsymbol\theta\rVert^2$ (§8.2.3) treats all coefficients as comparable, so without scaling it would essentially forbid the model from using latitude at all, while barely constraining the salary coefficient. Standardisation is what makes the regulariser mean the same thing in every direction.

The notation for the rest of Part II

• $N$ examples, indexed $n = 1, \dots, N$. Each example $\mathbf{x}_n$ is a $D$-dimensional vector.
• $D$ features, indexed $d = 1, \dots, D$.
• Supervised learning means each example carries a label $y_n$ — also called a target, response variable, or annotation.
• The dataset is $\{(\mathbf{x}_1, y_1), \dots, (\mathbf{x}_N, y_N)\}$, and the examples stack into $\mathbf{X} \in \mathbb{R}^{N\times D}$.

The book flags that this row-per-example orientation “originates from the database community”, and that Chapter 10 will find it more convenient to treat examples as columns. Worth knowing before you hit a transpose that looks wrong.

§8.1.2 Models as functions

A predictor is a function that takes a feature vector and returns an output. Taking the output to be a real scalar:

$$f : \mathbb{R}^D \to \mathbb{R}$$

The book restricts to linear functions, and says why: it strikes “a good balance between the generality of the problems that can be solved and the amount of background mathematics that is needed”. So:

$$f(\mathbf{x}) = \boldsymbol\theta^\top\mathbf{x} + \theta_0$$

for unknown $\boldsymbol\theta$ and $\theta_0$. Chapters 2 and 3 are enough to state this precisely — no functional analysis required.

The unit-feature trick, Equations 8.4 and 8.5

That $\theta_0$ is a nuisance: it makes the model affine rather than linear, so it does not fit the matrix machinery cleanly. The fix is to append a constant feature:

$$\mathbf{x}_n = [1,\ x_n^{(1)},\ x_n^{(2)},\ \dots,\ x_n^{(D)}]^\top, \qquad \boldsymbol\theta = [\theta_0,\ \theta_1,\ \dots,\ \theta_D]^\top$$

and then

$$f(\mathbf{x}_n, \boldsymbol\theta) = \boldsymbol\theta^\top\mathbf{x}_n \qquad\text{is identical to}\qquad f(\mathbf{x}_n, \boldsymbol\theta) = \theta_0 + \sum_{d=1}^{D}\theta_d x_n^{(d)}$$

The predictor now maps $\mathbb{R}^{D+1} \to \mathbb{R}$. The intercept has become an ordinary coefficient, which is convenient for the algebra and a genuine trap for regularisation: a penalty $\lVert\boldsymbol\theta\rVert^2$ now shrinks the intercept too, and there is usually no reason to want that. The book’s own margin note is that “affine functions are often referred to as linear functions in machine learning”, which is the same conflation seen from the vocabulary side.

§8.1.3 Models as probability distributions

The motivation is one sentence: data is usually noisy observations of some true underlying effect, and we would like predictors that “express some sort of uncertainty”. Chapter 6 is the language for that.

So instead of a single function, consider a distribution over possible outputs. The book restricts to distributions with finite-dimensional parameters, which is what lets it avoid stochastic processes and random measures, and notes that even this “already allow[s] for a rich class of models”.

The practical difference is exactly one question. With a function you can ask “what is the prediction at $x = 60$?” With a distribution you can also ask “how confident is that?” — and get an answer that is a number rather than a feeling.

§8.1.4 Learning is finding parameters

The goal: find a model and its parameters such that the resulting predictor does well on unseen data. Three conceptually distinct phases:

1. Prediction or inference — apply a trained predictor to new data. Model and parameters are fixed.
2. Training or parameter estimation — adjust the model using training data.
3. Hyperparameter tuning or model selection — choose among models.

For training there are two strategies, and this split organises the rest of the chapter:

• Find a point estimate. One best parameter vector. Works for both kinds of predictor. §8.2 (empirical risk minimization) and §8.3 (maximum likelihood, MAP).
• Bayesian inference. Keep a whole distribution over parameters. Needs a probabilistic model. §8.4.

The book’s own warning about “inference”

Verbatim: “there is no agreed upon naming for the different algorithmic phases. The word ‘inference’ is sometimes also used to mean parameter estimation of a probabilistic model, and less often may be also used to mean prediction for non-probabilistic models.”

So “inference” can mean phase 1 or phase 2 depending on who is speaking. In this module it means prediction with a probabilistic model, matching the book’s primary usage — but when reading anything else, check.

Worked example by hand

Fit the book’s own data. Table 8.2’s two rightmost columns are age and salary:

$$x = (36,\ 47,\ 26,\ 68,\ 33), \qquad y = (89.563,\ 123.543,\ 23.989,\ 138.769,\ 113.888)$$

Step 1: build the design matrix. Using the unit-feature trick with $D = 1$:

$$\boldsymbol\Phi = \begin{bmatrix}1 & 36\\ 1 & 47\\ 1 & 26\\ 1 & 68\\ 1 & 33\end{bmatrix}$$

Step 2: the normal equations. The least-squares solution satisfies $\boldsymbol\Phi^\top\boldsymbol\Phi\,\boldsymbol\theta = \boldsymbol\Phi^\top\mathbf{y}$. With $N = 5$, $\sum x_n = 210$ so $\bar{x} = 42$, and $\sum x_n^2 = 1296 + 2209 + 676 + 4624 + 1089 = 9894$:

$$\boldsymbol\Phi^\top\boldsymbol\Phi = \begin{bmatrix}5 & 210\\ 210 & 9894\end{bmatrix}$$

The determinant is $5 \times 9894 - 210^2 = 49470 - 44100 = 5370$. And $\sum y_n = 489.752$, $\sum x_n y_n = 22703.093$, so

$$\boldsymbol\theta = \frac{1}{5370}\begin{bmatrix}9894 & -210\\ -210 & 5\end{bmatrix}\begin{bmatrix}489.752\\ 22703.093\end{bmatrix} = \begin{bmatrix}8.807355\\ 2.122453\end{bmatrix}$$

Step 3: read it. The slope is $2.122453$ thousand per year of age, and the intercept is $8.807355$ thousand — a notional salary at age zero, which is why intercepts are usually not worth interpreting.

Step 4: predict at 60.

$$f(60) = 8.807355 + 2.122453 \times 60 = 136.154562$$

Step 5: the same fit as a distribution. The residuals are

$$\mathbf{y} - \boldsymbol\Phi\boldsymbol\theta, \qquad \text{empirical risk} = \frac1N\sum_n r_n^2 = 655.523816$$

so the residual standard deviation is $\sqrt{655.523816} = 25.603199$. The distributional answer at age $60$ is $\mathcal{N}(136.1546,\ 25.6032^2)$, giving a one-standard-deviation interval of $[110.551,\ 161.758]$ and a two-standard-deviation interval of $[84.948,\ 187.361]$.

That interval is enormous — nearly $\pm 40\%$ — and it should be, from five data points. The function view returns $136.154562$ and says nothing. That difference is the entire content of §8.1.3.

Figure 8.2’s line is not this fit

The book’s Figure 8.2 caption says the drawn function gives $f(60) = 100$. The least-squares fit to Table 8.2’s five points gives $136.154562$. There is no contradiction: Figure 8.2’s caption calls it an “example function” — it is illustrative, drawn to show what a predictor looks like, not fitted. Figure 8.5 is the one that shows the actual maximum likelihood fit. Worth checking rather than assuming, since the two figures share axes and look similar.

See it move

The four encoding decisions are easier to feel than to read about. Change one and watch what the model can and cannot then express:

sketch Four decisions, and what each one costs you p5.js Pause Restart

Step through the four encoding decisions that turn Table 8.1 into Table 8.2. Each panel shows what the column looked like before, what it becomes, and what information is now unrecoverable. The last knob toggles column standardisation and reports the spread ratio it fixes.

From scratch

data_models_learning.py

import numpy as np
 
# Table 8.1, exactly as the book prints it.
RAW = [
    ("Aditya",    "M", "MSc",   "W21BG",   36,  89563),
    ("Bob",       "M", "PhD",   "EC1A1BA", 47, 123543),
    ("Chloe",     "F", "BEcon", "SW1A1BH", 26,  23989),
    ("Daisuke",   "M", "BSc",   "SE207AT", 68, 138769),
    ("Elisabeth", "F", "MBA",   "SE10AA",  33, 113888),
]
 
# The four decisions that produce Table 8.2.
GENDER = {"M": -1, "F": +1}                      # decision 2
DEGREE = {"BSc": 1, "BEcon": 1, "MSc": 2, "MBA": 2, "PhD": 3}   # decision 3
POSTCODE = {                                     # decision 4, domain knowledge
    "W21BG":   (51.5073, 0.1290),
    "EC1A1BA": (51.5074, 0.1275),
    "SW1A1BH": (51.5071, 0.1278),
    "SE207AT": (51.5075, 0.1281),
    "SE10AA":  (51.5074, 0.1278),
}
 
rows = []
for name, g, deg, pc, age, salary in RAW:      # decision 1: drop the name
    lat, lon = POSTCODE[pc]
    rows.append([GENDER[g], DEGREE[deg], lat, lon, age, salary / 1000.0])
X_full = np.array(rows)
print("Table 8.2 reconstructed, shape", X_full.shape)
for r in X_full:
    print("  [%+d, %d, %.4f, %.4f, %d, %.3f]"
          % (r[0], r[1], r[2], r[3], r[4], r[5]))
 
# --- N and D, and the supervised split ------------------------------------
N, D = X_full.shape[0], X_full.shape[1] - 1
print(f"\nN = {N} examples, D = {D} features (salary is the label y)")
x = X_full[:, 4]                                  # age
y = X_full[:, 5]                                  # salary in thousands
 
# --- Equation 8.2: a predictor as a function ------------------------------
# Concatenate a unit feature so the affine model is a linear one (Eq 8.4/8.5).
Phi = np.column_stack([np.ones(N), x])
theta = np.linalg.lstsq(Phi, y, rcond=None)[0]
print(f"\ntheta = [{theta[0]:.6f}, {theta[1]:.6f}]   "
      f"(intercept, slope per year)")
pred60 = theta @ np.array([1.0, 60.0])
print(f"f(60) = {pred60:.6f}")
resid = y - Phi @ theta
print(f"empirical risk (Eq 8.6, squared loss) = {np.mean(resid ** 2):.6f}")
print(f"residual standard deviation           = "
      f"{np.sqrt(resid @ resid / N):.6f}")
 
# --- Section 8.1.3: the same predictor as a distribution -----------------
sigma = np.sqrt(resid @ resid / N)
print(f"\nas a distribution: p(y | x=60) = N({pred60:.4f}, {sigma:.4f}^2)")
for k in (1, 2):
    print(f"  {k} sd interval: [{pred60 - k * sigma:8.3f}, "
          f"{pred60 + k * sigma:8.3f}]")
print("with five data points the honest interval is very wide, which is")
print("exactly the information the function view discards.")
 
# --- the book's own advice: standardise every column ---------------------
print("\nSection 8.1.1's advice, applied to all six columns:")
mu, sd = X_full.mean(axis=0), X_full.std(axis=0)
Z = (X_full - mu) / sd
names = ["gender", "degree", "latitude", "longitude", "age", "salary"]
print(f"{'column':>10} {'raw mean':>12} {'raw sd':>12} "
      f"{'scaled mean':>13} {'scaled sd':>11}")
for k, nm in enumerate(names):
    print(f"{nm:>10} {mu[k]:>12.6f} {sd[k]:>12.6f} "
          f"{Z[:, k].mean():>13.2e} {Z[:, k].std():>11.6f}")
print("\nlatitude had a spread of %.6f and salary %.4f -- a ratio of %.0f."
      % (sd[2], sd[5], sd[5] / sd[2]))
print("Any penalty on the size of theta treats those two as comparable")
print("unless the columns are scaled first, which is why the advice exists.")

text

Table 8.2 reconstructed, shape (5, 6)
  [-1, 2, 51.5073, 0.1290, 36, 89.563]
  [-1, 3, 51.5074, 0.1275, 47, 123.543]
  [+1, 1, 51.5071, 0.1278, 26, 23.989]
  [-1, 1, 51.5075, 0.1281, 68, 138.769]
  [+1, 2, 51.5074, 0.1278, 33, 113.888]
 
N = 5 examples, D = 5 features (salary is the label y)
 
theta = [8.807355, 2.122453]   (intercept, slope per year)
f(60) = 136.154562
empirical risk (Eq 8.6, squared loss) = 655.523816
residual standard deviation           = 25.603199
 
as a distribution: p(y | x=60) = N(136.1546, 25.6032^2)
  1 sd interval: [ 110.551,  161.758]
  2 sd interval: [  84.948,  187.361]
with five data points the honest interval is very wide, which is
exactly the information the function view discards.
 
Section 8.1.1's advice, applied to all six columns:
    column     raw mean       raw sd   scaled mean   scaled sd
    gender    -0.200000     0.979796     -4.44e-17    1.000000
    degree     1.800000     0.748331     -8.88e-17    1.000000
  latitude    51.507340     0.000136      1.11e-17    1.000000
 longitude     0.128040     0.000516      2.15e-14    1.000000
       age    42.000000    14.656057      2.22e-17    1.000000
    salary    97.950400    40.288419     -2.89e-16    1.000000
 
latitude had a spread of 0.000136 and salary 40.2884 -- a ratio of 297010.
Any penalty on the size of theta treats those two as comparable
unless the columns are scaled first, which is why the advice exists.

On real data

figure Turning a table into vectors is a sequence of modelling decisions matplotlib

Each of the four decisions is a claim about the world, and each is irreversible downstream. Nothing later can recover the postcode from a latitude, or learn that an MBA is not an MSc.

figure One fit, two kinds of answer matplotlib

The function view returns 136.154562 and stops. The distribution view returns the same mean with a standard deviation of 25.603199, so its two-sigma interval spans 84.9 to 187.4 — which is the honest answer from five points.

figure Section 8.1.4's three phases, and what is fixed in each matplotlib

The three phases consume different slices of the data and are all called learning. The book warns explicitly that the word inference is used for at least two of them.

Reading the plot

The first figure counts what a data-loading script actually does. Look at the “lost” column of the four decisions rather than the “why” column. Decision 1 loses nothing predictive. Decision 2 loses nothing at all — until a regulariser appears, at which point the choice between $\{-1, +1\}$ and $\{0, 1\}$ changes the geometry of the penalty. Decision 3 loses the most: it asserts a total order on degrees and an equal spacing and two specific equalities (BEcon $=$ BSc, MBA $=$ MSc), and none of those is recoverable later. Decision 4 loses the postcode, and could only be made by someone who knew what a postcode is.

The point is not that any of these is wrong. It is that they are decisions, and a table of numbers carries no record of having been decided. By the time the data reaches §8.2 it looks like a given.

The second figure is §8.1.2 against §8.1.3 on identical numbers. Same five points, same line, $\boldsymbol\theta = (8.807355,\ 2.122453)$. On the left the answer at age $60$ is $136.154562$ — a number with no error bar and no way to attach one, because a function has no vocabulary for doubt.

On the right the answer is a Gaussian with the same mean and standard deviation $25.603199$. Look at how wide that is: the two-sigma interval runs from $84.948$ to $187.361$, more than a factor of two from bottom to top. That is not a defect of the distributional view — it is the truth about a fit to five points, and the function view’s crisp $136.154562$ was hiding it. Which of the two you want depends entirely on whether anything downstream is going to make a decision with the number.

One honest note on the residual scale: I computed it as $\sqrt{\sum r_n^2 / N}$ with $N = 5$, the maximum likelihood estimate that §8.3.1 will derive. It is biased low, because it divides by $N$ rather than by $N - 2$ for the two fitted parameters. Using $N - 2$ gives $33.053588$, an even wider interval. The point stands either way, and gets stronger.

The third figure is the map for the rest of Part II. Read it from the bottom up, because that is the order of increasing freedom: at prediction time everything is fixed; at training time the parameters move but the model class does not; at model-selection time the class itself moves. Each layer wraps the one below, which is exactly why §8.6.1’s nested cross-validation has two loops — one per layer of freedom.

The naming note in the caption is not pedantry. If a paper says “inference took 4 ms”, that is phase 1. If it says “variational inference”, that is phase 2. The book’s own remark that there is “no agreed upon naming” is the safest thing to carry.

From the book

This page covers §8.1 of Mathematics for Machine Learning by Deisenroth, Faisal and Ong (draft of 2019-12-11), including §8.1.1 (data as vectors), §8.1.2 (models as functions), §8.1.3 (models as probability distributions) and §8.1.4 (learning is finding parameters). Table 8.1 and Table 8.2 are reproduced verbatim; the tidy-data reference is Wickham (2014) and Codd (1990), and the data-science pointers are Stray (2016) and Adhikari and DeNero (2018). Equation 8.1 is $f : \mathbb{R}^D \to \mathbb{R}$ and 8.2 is the affine predictor. Figures 8.1, 8.2 and 8.3 are the toy dataset, the example function with $f(60) = 100$, and the same function with predictive uncertainty drawn as a Gaussian. The three algorithmic phases and the remark that “there is no agreed upon naming” are from §8.1.4. The advice to shift and scale every column to zero mean and unit variance is the book’s. The reconstruction of Table 8.2 from Table 8.1, the fitted $\boldsymbol\theta = (8.807355, 2.122453)$, the value $f(60) = 136.154562$, the residual scale $25.603199$ and the $297{,}010$ spread ratio are this module’s additions.

Pitfalls

Treating the numerical table as the data

Table 8.2 looks like a fact and is the output of four decisions. The one with the most content is the degree encoding: $1/2/3$ asserts an order, equal spacing, and that BEcon $=$ BSc and MBA $=$ MSc. None of that is recoverable from the numbers, and no diagnostic will flag it — the model will simply behave as though those claims are true.

Regularising without standardising

Measured on Table 8.2: latitude has spread $0.000136$, salary has spread $40.288419$, a ratio of $297{,}010$. A penalty $\lVert\boldsymbol\theta\rVert^2$ treats all coefficients as comparable, so without scaling it effectively forbids the model from using latitude while barely touching the salary coefficient. The book’s advice to standardise every column is what makes the penalty isotropic.

Letting the unit-feature trick regularise your intercept

Equations 8.4 and 8.5 absorb $\theta_0$ into $\boldsymbol\theta$, which is exactly what you want for the algebra. It also means $\lVert\boldsymbol\theta\rVert^2$ now includes $\theta_0^2$, shrinking the intercept toward zero — and there is rarely a reason to prefer a model whose prediction at the origin is small. In practice the intercept is excluded from the penalty; nothing in the notation reminds you.

Reading an intercept as a quantity

The fitted intercept here is $8.807355$ thousand, nominally a salary at age zero. It is not a prediction about newborns; it is where the line happens to cross an axis far outside the data, which ran from $26$ to $68$. Intercepts are usually reparameterisation artifacts, and centring the inputs turns this one into the mean salary instead.

Saying “inference” without saying which phase

The book states there is no agreed convention. “Inference” most often means prediction with a probabilistic model, sometimes means parameter estimation, and occasionally means prediction with a non-probabilistic model. Three phases, one word — and the ambiguity survives into library APIs.

Assuming row-per-example everywhere

§8.1.1 stacks examples as rows, $\mathbf{X} \in \mathbb{R}^{N\times D}$, following database convention. The book notes Chapter 10 finds columns more convenient. A transpose that looks like a bug is often this convention change.

Compare

	model as a function	model as a distribution
the object	$f : \mathbb{R}^D \to \mathbb{R}$	$p(y \mid \mathbf{x}, \boldsymbol\theta)$
the book	§8.1.2, Eq 8.2	§8.1.3
answer at $x = 60$	$136.154562$	$\mathcal{N}(136.1546,\ 25.6032^2)$
can express uncertainty	no	yes
trained by	empirical risk minimization, §8.2	maximum likelihood or Bayes, §8.3, §8.4
regularisation appears as	a penalty term, §8.2.3	a prior, §8.3.2
needs a noise model	no	yes
prediction is called	prediction	inference

phase	what is fixed	what is free	data it consumes	where
prediction / inference	model and parameters	the input only	one unseen point	Ch 9–12
training / parameter estimation	the model class	the parameters	the training set	§8.2, §8.3, §8.4
model selection	nothing	the class and hyperparameters	training + validation	§8.6

pch.quizTag Do you know what the vocabulary is hiding?

1. Table 8.2 encodes Degree as 1, 2, 3. What does that assert?

   A Nothing; it is just a label for each category B An order on degrees, an equal spacing between consecutive levels, and that BEcon equals BSc and MBA equals MSc C That degree is the most important feature D That there are exactly three degree types in the world
   pch.quizShowAnswer

   B — An order on degrees, an equal spacing between consecutive levels, and that BEcon equals BSc and MBA equals MSc — An ordinal encoding is three claims at once and all of them are domain claims. None is recoverable from the numbers afterwards, and nothing will flag them — a linear model will simply behave as though the step from bachelor to master equals the step from master to doctorate.

2. Why does the book advise standardising every column to zero mean and unit variance?

   A To make gradient descent converge, which is the only reason B Because a penalty on the size of theta treats all coefficients as comparable, and Table 8.2's column spreads differ by a factor of 297010 C Because probability distributions require standardised inputs D To remove outliers from the data
   pch.quizShowAnswer

   B — Because a penalty on the size of theta treats all coefficients as comparable, and Table 8.2's column spreads differ by a factor of 297010 — Latitude has standard deviation 0.000136 and salary 40.288419. An isotropic penalty on theta would effectively forbid the model from using latitude while barely constraining salary. Conditioning also improves, but the regularisation argument is the one that changes the answer rather than the speed.

3. What exactly does the unit-feature trick of Equations 8.4 and 8.5 buy, and what does it cost?

   A It buys nothing; it is purely notational B It turns an affine model into a linear one so matrix algebra applies — at the cost that a norm penalty on theta now shrinks the intercept too C It removes the need for an intercept entirely D It makes the model nonlinear
   pch.quizShowAnswer

   B — It turns an affine model into a linear one so matrix algebra applies — at the cost that a norm penalty on theta now shrinks the intercept too — Appending a constant feature makes theta-transpose x identical to the affine form, which is what lets the whole of Chapters 2 and 3 apply. The side effect is that theta-0 is now an ordinary coefficient, so the L2 penalty of Section 8.2.3 shrinks it — and there is rarely a reason to prefer a small intercept.

4. The fit to Table 8.2 gives f(60) = 136.154562 with residual standard deviation 25.603199. What is the honest reading?

   A The salary at age 60 is 136.15 thousand B The point prediction is 136.15 but the two-sigma interval spans 84.9 to 187.4, so the estimate is barely informative C The model is wrong, since the interval is so wide D The residual standard deviation is irrelevant to prediction
   pch.quizShowAnswer

   B — The point prediction is 136.15 but the two-sigma interval spans 84.9 to 187.4, so the estimate is barely informative — The width is not a defect, it is the truth about a two-parameter fit to five points — and dividing by N minus 2 rather than N makes it wider still, at 33.054. The function view's crisp 136.154562 was concealing exactly this.

5. In the book's terminology, what does 'inference' most often mean?

   A Parameter estimation, always B Prediction with a probabilistic model — but the book warns there is no agreed convention and it is also used for parameter estimation C Model selection D The training phase of any model
   pch.quizShowAnswer

   B — Prediction with a probabilistic model — but the book warns there is no agreed convention and it is also used for parameter estimation — Section 8.1.4 states verbatim that there is no agreed naming: inference usually means prediction with a probabilistic model, is sometimes used for parameter estimation, and occasionally for prediction with a non-probabilistic one. Three phases share the word, and the ambiguity survives into library APIs.

🧪 Try It Yourself

Exercise 1 – Reconstruct Table 8.2

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Fit the book’s own example

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Why standardisation matters

pch.exerciseTag pch.exerciseTitle

Exercise 4 – A function cannot express doubt

pch.exerciseTag pch.exerciseTitle

Exercise 5 – The intercept is not a quantity

pch.exerciseTag pch.exerciseTitle

Recall card

• Three components: data, models, learning. The guiding question is “what do we mean by good models?”, and the book’s answer is that good models perform well on UNSEEN data.
• Data as vectors is a sequence of decisions, not a format conversion. Table 8.1 to Table 8.2 takes four: drop the name, encode gender as minus one and plus one, encode degree as an ordinal 1 / 2 / 3, and turn a postcode into a latitude and longitude.
• The degree encoding carries the most content: it asserts an order, an equal spacing, and that BEcon equals BSc and MBA equals MSc. None of that is recoverable from the numbers.
• Standardise every column, because a penalty on the size of theta treats all coefficients as comparable. Measured on Table 8.2, the widest and narrowest column spreads differ by a factor of 297010.
• N examples indexed by n, D features indexed by d, examples stacked as ROWS in X. Chapter 10 switches to columns, so a surprising transpose is often just that.
• Equation 8.2 is a model as a function, restricted to linear ones because that keeps the required mathematics inside Chapters 2 and 3.
• The unit-feature trick, Equations 8.4 and 8.5, appends a constant feature so the affine model becomes linear. The cost: the intercept is now an ordinary coefficient and an L2 penalty shrinks it, which you almost never want.
• Section 8.1.3 is a model as a distribution, and it buys exactly one thing: the question “how confident are you?” becomes answerable.
• The book’s own example fitted: theta is (8.807355, 2.122453), f(60) is 136.154562, and the residual scale is 25.603199 — so the two-sigma interval at age 60 spans 84.9 to 187.4, or 75 percent of the prediction. Figure 8.2’s line is illustrative, not this fit.
• An intercept is not a quantity. Centring the input turns 8.807355 into 97.950400, which is the mean salary — same model, same slope, same predictions.
• Three algorithmic phases: prediction (everything fixed), training (parameters free), model selection (the class free). They consume different slices of the data and Section 8.6.1’s nested cross-validation has one loop per layer.
• Two training strategies: find a point estimate (Sections 8.2, 8.3) or do Bayesian inference (Section 8.4). Only the second needs a probabilistic model.
• “Inference” is ambiguous and the book says so. It usually means prediction with a probabilistic model, sometimes parameter estimation, occasionally prediction with a non-probabilistic model.

Next: the first of the two frameworks — learning without ever writing down a probability. Empirical Risk Minimization