Phase 7 - Reinforcement Learning

In every phase so far the data arrived first and the model came second. Reinforcement learning inverts that: the agent’s own choices decide what data it ever sees. An action not taken teaches nothing, and a policy that settles early will keep confirming whatever it settled on.

That single property causes almost every result in this phase, including the ones that contradict the standard story.

What was measured

Everything ran on this machine. There is no gymnasium installed and nothing may be downloaded, so all three environments are hand-written in scripts/figures/dl/_rl.py — a k-armed bandit, a stochastic gridworld solvable exactly by value iteration, and a vectorised CartPole using the same constants as the classic implementation. CartPole validates against it: a random policy survives 23.09 steps here against the reference’s ~22.

Page	The claim being tested	Result
Exploration vs exploitation	Greedy is worse than ε-greedy	True on average (520.92 vs 224.74) — but greedy’s seeds ranged 0.00 to 2394.40
Introduction to RL	The discount factor is a technical detail	False — it switches the optimal route and moves the goal rate 0.7925 → 0.8900
Q-learning and DQN	Replay and target networks each help	Only together: 144.20 against 53.45 for either alone and 48.81 for neither
Policy gradients	A baseline improves learning	It cut the gradient norm 35× and changed the score by 0.27
Actor-critic and PPO	PPO is more stable than REINFORCE	Only past 4 reuse steps — at 16 the unclipped worst seed scored 9.2, below random. Sample efficiency: 170 episodes against 544

figure 200 bandits per strategy, 1000 pulls each matplotlib

Optimistic initialisation has the lowest mean regret at 137.07 and UCB is second at 152.03 — but the error bars reverse the ranking's reliability. UCB's spread across seeds is 31.49 with a worst case of 244.06, while optimistic initialisation varies by 243.88 and can reach 1278.93. If you must be right on one attempt rather than on average, those are different recommendations.

figure 6,000 frames across 16 parallel environments, 3 seeds each matplotlib

The two mechanisms are not independently useful here — each alone lands at 53.45, barely above the 48.81 of using neither. Together they reach 144.20. The error bars carry the other half of the story: replay-only is remarkably consistent (sd 1.84) because it reliably plateaus, while target-only ranges from 25.47 to 105.12 depending on the seed.

figure Every claim this phase set out to test matplotlib

Collected from the runs behind each page's own figures rather than measured afresh, so every bar is traceable to the page it names. Bar length is the log of the effect size, because the effects span from 0.0014 to 5,376 — the number that matters is printed on each bar. Across all nine phases, 34 of 54 claims were contradicted outright, 11 held at a cost that was worth stating, and 9 held as advertised.

The thread running through all four

diagram Diagram mermaid

  flowchart TD
  E["the agent chooses what data it sees"] --> B["bandit: greedy stops looking
0.32 optimal actions"]
  E --> G["gridworld: Q-learning never walks
the safe corridor, 0.9625 shortcut"]
  E --> D["DQN: correlated samples
need a replay buffer"]
  E --> P["REINFORCE: every return positive,
so everything is reinforced"]
  E --> A["PPO: reuse a batch four times,
but only inside a trust region"]
  A --> M
  B --> M["measure against a known optimum"]
  G --> M
  D --> M
  P --> M

Each page attacks the same problem at a different scale, and each one needs an external reference to be interpretable:

• The bandit knows its best arm, so regret is exact.
• The gridworld is small enough for value iteration, so the optimal policy is known.
• CartPole has a 200-step cap and a measured random baseline of 23.09 — which is what makes the unclipped PPO seed that scored 9.2 legible as a catastrophe rather than a low number.

Without those references, every curve in this phase would merely go up.

sketch How often the standard story survived p5.js Pause Restart

Step through the phases. Each bar splits the claims that phase tested into contradicted, held at a price, and held as advertised - the totals are summed live.

What went differently than expected

Three results are worth carrying forward, because they are the kind that a plot alone would hide.

Q-learning found a worse policy than the optimum and looked converged. On the gridworld it took the risky shortcut in 96.25% of episodes where the exact solution takes it in 7.75%, scoring 0.5497 against 0.6568. Nothing about its learning curve says so — the failure is that ε-greedy almost never traverses a 12-step corridor, so the better route was never evaluated.

Low regret can mean an easy problem. ε-greedy’s regret was lowest on the hardest bandit (99.87 with arms close together, 414.38 with them far apart) because a wrong choice costs little when the arms are nearly identical. The optimal-action share tells the opposite, correct story.

Variance reduction is not the same as better learning. REINFORCE’s baseline did exactly what theory says to the gradient — 1.6992 → 0.0478 — and the final score moved 0.27. Separately, smaller batches beat larger ones at a fixed episode budget (200.00 against 173.04) despite having measurably noisier gradients.

Two engineering facts that decide whether any of this runs

Both were measured while building these pages, and both matter more than any hyperparameter here:

Way to run a batch through a network	Time per call
model.predict(x)	~470 ms
model(x) eager	~28 ms
Traced tf.function	~1.2 ms

An RL loop makes several forward passes per environment frame, so that 390× gap is the difference between a page that runs in minutes and one that does not run at all. The same logic applies to the environment: stepping 16 CartPoles together costs one model call per timestep instead of sixteen, measured at roughly 16× faster for REINFORCE’s episode collection.

sketch The two mechanisms, toggled p5.js Pause Restart

Click either switch. The bar shows the measured final episode length for that combination — the point being that neither switch does much alone.

Reading order

1. Exploration vs exploitation — the dilemma with everything else stripped away, and the reason a single run can never rank two strategies.
2. Introduction to reinforcement learning — add states, and a discount factor that decides between a gamble and a detour.
3. Q-learning and deep Q-networks — the tabular update, why a network breaks it, and an ablation of the two mechanisms that repair it.
4. Policy gradients — skip the value function entirely and pay for it in variance.
5. Actor-critic and PPO — put a learned value function back in as a baseline, then make it safe to take four gradient steps on one batch instead of one.

What this phase does not cover

• Pixel-based environments. Everything here uses low-dimensional state vectors, so no convolutional encoder is involved and none of the Atari-scale engineering appears.
• Long-horizon or sparse-reward tasks. CartPole’s reward is dense and its episodes are capped at 200 steps, which is precisely why the REINFORCE baseline had no room to show a benefit.

Each page states its own budget beside its numbers, so nothing here needs to be taken on trust.

Next

Start with Exploration vs Exploitation — ten levers, a thousand pulls, and the measurement that every later page depends on.

pch.quizTag pch.quizDefaultTitle

1. A DQN with a replay buffer alone reached 53.45 and with a target network alone reached 53.45, against 48.81 for neither and 144.20 for both. What does that pattern mean?

   A The two mechanisms do the same job, so either one is enough B The measurements are too noisy to interpret C Each mechanism only pays off once the other has removed a different failure — their effects are not additive D The target network is redundant
   pch.quizShowAnswer

   C — Each mechanism only pays off once the other has removed a different failure — their effects are not additive

2. Every environment in this phase is hand-written in `_rl.py` rather than imported from a library. Why does the phase check that a random CartPole policy survives 23.09 steps?

   A To confirm the random number generator is seeded B Because a hand-written environment has to be validated against the reference implementation before any result measured in it means anything C Because 23 steps is the target score D To measure how fast the environment runs
   pch.quizShowAnswer

   B — Because a hand-written environment has to be validated against the reference implementation before any result measured in it means anything

3. ε-greedy scored its LOWEST regret (99.87) on the bandit whose arms were closest together, and its highest (414.38) when they were far apart. What is going on?

   A Close-together arms are genuinely easier to solve B Regret measures reward lost per wrong choice, and when the arms are nearly identical a wrong choice costs almost nothing — low regret here means low stakes, not good behaviour C The strategy adapts to the difficulty automatically D The measurement is wrong
   pch.quizShowAnswer

   B — Regret measures reward lost per wrong choice, and when the arms are nearly identical a wrong choice costs almost nothing — low regret here means low stakes, not good behaviour

4. Q-learning on the gridworld took the risky shortcut in 96.25% of episodes where the exact solution takes it in 7.75%, and its learning curve looked converged. What caused it?

   A The learning rate was too high B The discount factor was wrong C ε-greedy almost never strings together the 12 steps needed to traverse the safe corridor, so the better route was never evaluated — the agent's own policy decided what data existed D Not enough episodes
   pch.quizShowAnswer

   C — ε-greedy almost never strings together the 12 steps needed to traverse the safe corridor, so the better route was never evaluated — the agent's own policy decided what data existed

5. `model.predict(x)` measured ~470 ms per call, eager `model(x)` ~28 ms, and a traced `tf.function` ~1.2 ms. Why does this decide whether an RL page exists at all?

   A It does not — it only affects the final benchmark table B An RL loop makes several forward passes per environment frame and needs thousands of episodes, so a 390× per-call gap is the difference between minutes and days C Because `predict` gives different answers D Because tracing improves accuracy
   pch.quizShowAnswer

   B — An RL loop makes several forward passes per environment frame and needs thousands of episodes, so a 390× per-call gap is the difference between minutes and days