What you’ll learn

• the agent, environment, action, observation, and reward vocabulary
• what a policy is, and why some policies are stochastic
• three ways to search for a good policy (brute force, genetic algorithms, gradients)
• how to create and step through an OpenAI Gym environment
• how to hand-code a simple baseline policy for CartPole
• the explore vs. exploit trade-off, and why some policies act randomly on purpose

The agent-environment-reward loop

In Reinforcement Learning, an agent observes an environment, takes an action, and receives a reward. It repeats this over and over, and its only goal is to maximize the total reward it collects — not to be “correct” on any single step, the way supervised learning is.

• agent — the decision-maker (a robot’s controller, a game-playing program, a thermostat)
• environment — everything the agent doesn’t directly control (the real world, a game simulator)
• observation — what the agent perceives about the environment’s current state
• action — what the agent is allowed to do
• reward — a number telling the agent how good or bad that step was

If you don’t mind a bit of anthropomorphism, positive rewards feel like pleasure and negative rewards feel like pain — the agent learns, by trial and error, to chase the first and avoid the second.

  flowchart LR
  Obs["Observation"] --> Policy["Policy (Agent)"]
  Policy --> Act["Action"]
  Act --> Env["Environment"]
  Env -->|"Reward"| Policy
  Env -->|"Next Observation"| Obs

Policy search

The algorithm an agent uses to pick actions is called its policy. A policy does not have to be a neural network, and it does not have to be deterministic — a policy that involves randomness is called a stochastic policy.

Take a robot vacuum whose only reward is how much dust it picks up in 30 minutes. A simple stochastic policy could be: move forward with probability pp each second, otherwise rotate left or right by a random angle. The only two parameters are pp and the rotation range rr. There are a few ways to search for good values of pp and rr:

• brute force — try many combinations, keep the best one (policy search)
• genetic algorithms — breed the best-performing policies across generations
• policy gradients (PG) — nudge the parameters in the direction that increases reward, using gradients

We’ll build up to policy gradients later in this phase. First, we need something to practice on — an environment.

OpenAI Gym: CartPole

OpenAI Gym is a toolkit that ships many simulated environments (Atari games, board games, physics simulations) so you can train and compare agents without writing a simulator yourself. Here’s the classic CartPoleCartPole environment: a cart that can be pushed left or right to balance a pole standing on top of it.

import gym
import numpy as np
 
env = gym.make("CartPole-v1")
obs = env.reset()
print("Initial observation:", obs)
print("Action space:", env.action_space)     # Discrete(2): 0 = left, 1 = right
 
# obs = [cart position, cart velocity, pole angle, pole angular velocity]
action = 1 if obs[2] > 0 else 0              # push toward the side the pole leans
obs, reward, done, info = env.step(action)
print("New observation:", obs)
print("Reward:", reward, "| Done:", done)
env.close()

import gym
import numpy as np
 
env = gym.make("CartPole-v1")
obs = env.reset()
print("Initial observation:", obs)
print("Action space:", env.action_space)     # Discrete(2): 0 = left, 1 = right
 
# obs = [cart position, cart velocity, pole angle, pole angular velocity]
action = 1 if obs[2] > 0 else 0              # push toward the side the pole leans
obs, reward, done, info = env.step(action)
print("New observation:", obs)
print("Reward:", reward, "| Done:", done)
env.close()

Every call to env.step(action)env.step(action) returns four things:

• obsobs — the new observation
• rewardreward — 1.01.0 for every step in CartPole, so the goal is simply to survive as long as possible
• donedone — TrueTrue once the pole has tilted too far, the cart left the screen, or you hit the step limit
• infoinfo — an environment-specific dictionary, often empty

import gym env = gym.make(“CartPole-v1”) obs = env.reset() obs array([-0.0126, -0.0016, 0.0421, -0.0018]) env.action_space Discrete(2)

A hardcoded baseline policy

Before reaching for a neural network, try the simplest thing that could work: push right when the pole leans right, push left when it leans left.

import gym
import numpy as np
 
env = gym.make("CartPole-v1")
 
def basic_policy(obs):
    angle = obs[2]
    return 0 if angle < 0 else 1
 
totals = []
for episode in range(500):
    episode_rewards = 0
    obs = env.reset()
    for step in range(200):
        action = basic_policy(obs)
        obs, reward, done, info = env.step(action)
        episode_rewards += reward
        if done:
            break
    totals.append(episode_rewards)
 
print("mean:", np.mean(totals), "std:", np.std(totals))
print("min:", np.min(totals), "max:", np.max(totals))
env.close()

import gym
import numpy as np
 
env = gym.make("CartPole-v1")
 
def basic_policy(obs):
    angle = obs[2]
    return 0 if angle < 0 else 1
 
totals = []
for episode in range(500):
    episode_rewards = 0
    obs = env.reset()
    for step in range(200):
        action = basic_policy(obs)
        obs, reward, done, info = env.step(action)
        episode_rewards += reward
        if done:
            break
    totals.append(episode_rewards)
 
print("mean:", np.mean(totals), "std:", np.std(totals))
print("min:", np.min(totals), "max:", np.max(totals))
env.close()

Even across 500 tries, this policy rarely balances the pole for more than ~70 steps — the cart oscillates more and more strongly until the pole tips over. Not great, but it’s a useful baseline before we bring in learning algorithms later in this phase.

Exploring vs. exploiting

The hardcoded policy above is deterministic — give it the same angle twice, and it makes the same choice both times. That’s fine for a two-line baseline, but it can never discover a rule better than the one you wrote by hand, because it never tries anything different.

A more capable policy — like the small neural network we’ll build in Policy Gradients (Intro) — instead outputs a probability for each action and then samples one at random. In CartPole, the network outputs pp, the probability of pushing left, so it pushes left with probability pp and right with probability 1 - p1 - p, rather than always doing whatever seems best right now.

Why bother with randomness? It lets the agent balance two competing needs:

• exploitation — take the action you currently believe is best, to actually collect reward
• exploration — try something else occasionally, in case it turns out to be even better

Géron’s analogy: imagine trying a restaurant for the first time. Every dish looks equally appealing, so you order one at random. If it’s great, you’ll probably raise the odds you’ll order it again — but not all the way to 100%, or you’ll never find out that another dish might have been even better. Push a policy too far toward “always exploit” too early and it gets stuck repeating a mediocre habit; keep it too random for too long and it never settles into acting on what it’s learned. We’ll meet a concrete, decaying version of this trade-off — the ε-greedy policy — on the next page.

Visualize it

Here’s the agent-environment-reward loop in action: a simplified CartPole where the pole tilts, the agent usually nudges the cart the way its policy suggests (blue), but every so often takes a random exploratory push instead (amber) — with the exploration rate epsilonepsilon decaying across episodes. The reward counter climbs every step it survives, until the pole tips too far and the episode resets.

Mini-checkpoint

If env.step()env.step() returns reward = 1.0reward = 1.0 on every single step no matter what the agent does, what should the agent actually be optimizing for?

• Not the reward value itself (it never changes) — but the number of steps it survives before donedone becomes TrueTrue.

🧪 Try It Yourself

Exercise 1 – Create and Reset an Environment

Exercise 2 – Write a Hardcoded Policy

Exercise 3 – Track Episode Rewards

Next

Continue to Q-Learning & Deep Q-Networks — learn how Markov decision processes formalize this loop, and how an agent can learn Q-values without ever being told the environment’s rules.