Phase 3 - Computer Vision with CNNs

Convolution exists because a Dense layer is the wrong shape for an image. On a 224×224×3 photograph, Dense(32) costs 4,816,928 parameters; a Conv2D(32, 3) costs 896 — and unlike the dense layer, that number does not change when the image does. Everything in this phase follows from weight sharing and locality.

Every number on these ten pages was measured on this machine, on a CPU, and the measurements are reproducible in minutes. Where a result contradicted the textbook claim, the page says so.

The pages

#	Page	The measured headline
421	Intro to CNNs	896 parameters against 4,816,928; convolution matched TensorFlow at 2.38e-07; two 3×3 layers beat one 5×5 (18 weights against 25)
422	Pooling & CNN Architecture	average 0.8740, max 0.8720, strided convolution last at 0.8455; Flatten→Dense held 803,072 of 880,936 weight bytes
423	Famous Architectures	VGG-ish 0.8800 wins; plain beat residual at 12 convolutions (0.8500 vs 0.8433)
423.5	Normalisation Beyond Batch	layer norm won at batches 4/16/64, batch norm overtook it at 256 — exactly where the statistic counts cross
424	Transfer Learning	fine-tuning beat scratch by +0.0014, freezing lost at every size, and a wrong learning rate cost 0.4026
424.5	Fine-Tuning and LoRA	LoRA reached 0.8911 training 2.80% of the weights against full fine-tuning’s 0.9385 — and training from scratch beat both at 0.9350
425	Data Augmentation	cost 0.1855 on canonical data and bought 0.1105 on rotated data — invariance, not accuracy
426	Image Segmentation	pixel accuracy 0.9737 against a do-nothing baseline of 0.8691; report IoU
427	Interpreting Convnets	activations grow 1.61 → 20.44 with depth; 4 of 64 channels silent; always run the random-weights check
428	Object Detection	32.8 : 1 cell imbalance; recall 0.8616 → 0.6799 while precision sat at 0.997
428.5	Vision Transformers	the ViT beat the convnet at 1,000 and 4,000 rows — and overfitted 4× harder

diagram Diagram mermaid

  flowchart TD
  A["421 Convolution
weight sharing, receptive field"] --> B["422 Pooling & the stack
downsampling, memory"]
  B --> C["423 Famous architectures
LeNet to Xception"]
  C --> D["423.5 Normalisation
batch, layer, group, instance"]
  D --> E["424 Transfer learning
freeze, fine-tune, probe"]
  E --> L["424.5 LoRA
low-rank adaptation"]
  L --> F["425 Augmentation
invariance, not accuracy"]
  F --> G["426 Segmentation
per-pixel labels, IoU"]
  G --> H["427 Interpretation
activations, Grad-CAM"]
  H --> I["428 Detection
grids, IoU, NMS"]
  I --> J["428.5 Vision transformers
patches and attention"]

figure 2,000 rows, 40 epochs — the same models on three test sets matplotlib

Read the un-augmented bars left to right: 0.763, 0.626, 0.596. That model loses 0.1370 the moment the test images move and 0.1665 when they rotate — it never had to cope with either. The medium-augmented model reads 0.695, 0.683, 0.707: almost flat. It gave up 0.068 on canonical data and bought back 0.1105 on rotated data. Neither model is better; they are optimised for different deployment conditions.

figure Garments to footwear and bags, same sensor, disjoint labels matplotlib

Three of the four curves are on top of each other, which is the result: at 250 target rows, training from scratch reaches 0.8628 and the best transfer strategy reaches 0.8642. The outlier is the low learning rate, and the right panel shows why — at 1e-4 the model is simply still training at epoch 12. That is not a property of fine-tuning, it is a budget mismatch, and it is the most common way transfer-learning comparisons get rigged.

figure Every claim this phase set out to test matplotlib

Collected from the runs behind each page's own figures rather than measured afresh, so every bar is traceable to the page it names. Bar length is the log of the effect size, because the effects span from 0.0014 to 5,376 — the number that matters is printed on each bar. Across all nine phases, 34 of 54 claims were contradicted outright, 11 held at a cost that was worth stating, and 9 held as advertised.

Four results that contradict the usual story

Each of these is measured on this data, at this scale — which is exactly the caveat that makes them useful rather than misleading.

1. Strided convolution came last. The “learnable downsampling replaces pooling” advice lost to plain average pooling by 0.0285. A fixed rule with no parameters beat a learned one at equal capacity.
2. Plain convolutions beat residual ones at 12 layers. Residual connections solve a problem that starts at depths this phase cannot afford; at 12 convolutions they cost 0.0067. Their direction was still visible — depth helped the residual family eight times more.
3. Augmentation hurt at every dataset size. Fashion-MNIST is centred and pose-normalised, and the un-augmented model was not overfitting. Augmentation encodes an invariance assumption; if the test set has no such variance, it is pure cost.
4. The ViT beat the convnet at 1,000 rows. “Transformers need enormous data” is a claim about 196+ tokens on 224×224 photographs. At 16 tokens on centred 28×28 greyscale, the convolutional prior is worth very little.

sketch How often the standard story survived p5.js Pause Restart

Step through the phases. Each bar splits the claims that phase tested into contradicted, held at a price, and held as advertised - the totals are summed live.

Three mistakes that were mine, and are now on the pages

Every one of these produced a plausible-looking number before it was caught.

• Batch normalisation’s default momentum of 0.99 needs roughly 460 batches for its moving averages to settle. A 376-batch run reported validation accuracy of 0.1540 with a training loss of 0.27. Lowering it to 0.9 fixed it, and the incident is written up on page 423.
• A ReLU placed before the Add in a residual block made the residual model worse than the plain one, because activations then grow additively with depth. The classic ordering is conv-BN-relu-conv-BN → Add → relu.
• A fixed epoch budget measured the budget rather than the treatment: augmented runs were still climbing at epoch 15 while plain runs had stopped. Page 425 reports the best epoch per run for exactly this reason.

sketch Augmentation is a bet on the test set p5.js Pause Restart

Drag to choose how much augmentation to apply, then click a test set. The measured accuracy changes sign depending on which distribution you deploy on.

What carries into the rest of the module

Idea from this phase	Where it comes back
weight sharing over a structured axis	1D convolution over time, Phase 4
layer normalisation, no batch statistics	every transformer block, Phases 4–6
residual connections around a sublayer	transformer blocks, diffusion U-Nets
the encoder–decoder hourglass with skips	U-Net diffusion models, Phase 6
attention over a set of tokens	the whole of Phase 5
“compare against the do-nothing baseline”	forecasting, detection, generation

From the book

Hands-On Machine Learning traces CNNs to Hubel and Wiesel’s 1958–59 finding that visual-cortex neurons respond to a small receptive field, and that simple detectors combine into complex ones deeper in. That hierarchy is what page 421 measures directly: a receptive field of 17 pixels after eight 3×3 layers at stride 1, and 511 at stride 2.

From the book

François Chollet frames architecture as the choice of a hypothesis space. A convolution encodes the prior that visual patterns are translation-invariant — page 421 measures that equivariance at exactly 0.00e+00 in the interior. Page 428.5 then removes the prior entirely and measures what it was worth: on this data, less than nothing.

pch.quizTag pch.quizDefaultTitle

1. On 224x224x3 input, `Dense(32)` costs 4,816,928 parameters and `Conv2D(32, 3)` costs 896. Which part of that comparison matters most?

   A That the convolution is smaller B That the convolution's count does not depend on the image size at all, because the same 3x3 kernel is reused at every position C That dense layers are obsolete D That 896 is a round number
   pch.quizShowAnswer

   B — That the convolution's count does not depend on the image size at all, because the same 3x3 kernel is reused at every position

2. Augmentation cost 0.1855 on canonical test data and bought 0.1105 on rotated test data. What is the correct conclusion?

   A Augmentation does not work B Augmentation encodes an invariance assumption, so it pays exactly when the deployment distribution contains that variance and costs accuracy when it does not C Fashion-MNIST is a bad dataset D The augmented model needed more epochs
   pch.quizShowAnswer

   B — Augmentation encodes an invariance assumption, so it pays exactly when the deployment distribution contains that variance and costs accuracy when it does not

3. Fine-tuning at learning rate 1e-4 reached only 0.4602 at 250 rows while 1e-3 reached 0.8642. Why is the low-rate result NOT evidence against fine-tuning?

   A Because 1e-4 is always wrong B Because the per-epoch curve shows it was still climbing at epoch 12 - it is a budget mismatch, not a property of the method C Because pretrained weights need higher rates by definition D Because the two runs used different data
   pch.quizShowAnswer

   B — Because the per-epoch curve shows it was still climbing at epoch 12 - it is a budget mismatch, not a property of the method

4. Batch normalisation's default momentum of 0.99 needs roughly 460 batches to settle, and a 376-batch run reported validation accuracy of 0.1540 with a training loss of 0.27. What was actually broken?

   A The model had not learned anything B Nothing in the model - the moving averages used at inference time had not converged, so training and validation were effectively using different normalisation statistics C The learning rate D The validation split
   pch.quizShowAnswer

   B — Nothing in the model - the moving averages used at inference time had not converged, so training and validation were effectively using different normalisation statistics

5. A vision transformer BEAT the convnet at 1,000 and 4,000 rows, contradicting 'transformers need enormous data'. What makes both statements true at once?

   A The measurement was wrong B The usual claim is about 196+ tokens on 224x224 photographs; at 16 tokens on centred 28x28 greyscale, the convolutional prior is worth very little, so there is far less for the transformer to relearn C Transformers are simply better D The convnet was undertrained
   pch.quizShowAnswer

   B — The usual claim is about 196+ tokens on 224x224 photographs; at 16 tokens on centred 28x28 greyscale, the convolutional prior is worth very little, so there is far less for the transformer to relearn