What you’ll learn

• what a language model actually is — a network that predicts a probability distribution over the next token, given the tokens that came before it
• the difference between character-level and word-level generation
• the text-generation loop: predict, sample, append, repeat
• why greedy sampling produces boring, repetitive text, and why pure random sampling produces gibberish
• the softmax temperature knob that sits between those two extremes, and how to implement it in a few lines of NumPy

A network that finishes your sentence

Every model you’ve built so far answers a question about existing data: is this email spam, what digit is this, what’s the sentiment of this review? A language model flips the task around — given some text so far, it predicts what’s likely to come next.

Concretely, feed the model the tokens "the cat sat on the""the cat sat on the" and it doesn’t output a single word. It outputs a full probability distribution over your entire vocabulary: maybe matmat gets 60%, floorfloor gets 15%, roofroof gets 5%, and thousands of other words share the rest. Tokens can be whole words (word-level) or single characters (char-level) — the mechanics are identical either way, just the vocabulary size changes (tens of thousands of words vs. a few dozen characters).

Once you have that distribution, generating new text is a loop: pick a token from the distribution, glue it onto the end of your text, feed the new, longer text back into the model, and repeat. Do that fifty times and you have fifty new tokens of machine-written text.

  flowchart LR
  P["Prompt: the cat sat on the"] --> M["Language model"]
  M --> D["Probability distribution over next token"]
  D --> S["Sampling strategy (temperature)"]
  S --> T["Sampled token: mat"]
  T --> A["Append to prompt"]
  A -->|"loop"| M

Building the training data: offset by one

You train a language model with completely ordinary supervised learning — the trick is how you build the labels. Take any sentence, and the target sequence is just the input sequence shifted over by one token:

import numpy as np
 
text = "the cat sat on the mat and the dog sat on the rug"
tokens = text.split()
vocab = sorted(set(tokens))
token_to_id = {t: i for i, t in enumerate(vocab)}
ids = np.array([token_to_id[t] for t in tokens])
 
# Input = tokens 0..N-1, Target = the SAME tokens shifted by one position
X = ids[:-1]
y = ids[1:]
 
print("vocab size:", len(vocab))
print("input tokens: ", tokens[:5])
print("target tokens:", tokens[1:6])

import numpy as np
 
text = "the cat sat on the mat and the dog sat on the rug"
tokens = text.split()
vocab = sorted(set(tokens))
token_to_id = {t: i for i, t in enumerate(vocab)}
ids = np.array([token_to_id[t] for t in tokens])
 
# Input = tokens 0..N-1, Target = the SAME tokens shifted by one position
X = ids[:-1]
y = ids[1:]
 
print("vocab size:", len(vocab))
print("input tokens: ", tokens[:5])
print("target tokens:", tokens[1:6])

vocab size: 9
input tokens:  ['the', 'cat', 'sat', 'on', 'the']
target tokens: ['cat', 'sat', 'on', 'the', 'mat']

vocab size: 9
input tokens:  ['the', 'cat', 'sat', 'on', 'the']
target tokens: ['cat', 'sat', 'on', 'the', 'mat']

At every position ii, the model sees tokens 0..i0..i and is trained to predict token i+1i+1. A tiny Keras model for this looks like an embedding layer feeding an RNN or a Transformer decoder, ending in a Dense(vocab_size, activation="softmax")Dense(vocab_size, activation="softmax") layer — the softmax is what turns raw scores into the probability distribution you sample from:

import tensorflow as tf
 
vocab_size = 40         # e.g. lowercase letters + space + punctuation
sequence_length = 20
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(vocab_size, 32, input_length=sequence_length),
    tf.keras.layers.LSTM(64, return_sequences=True),
    tf.keras.layers.Dense(vocab_size, activation="softmax"),
])
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam")
print("output shape (batch, timesteps, vocab):", model.output_shape)

import tensorflow as tf
 
vocab_size = 40         # e.g. lowercase letters + space + punctuation
sequence_length = 20
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Embedding(vocab_size, 32, input_length=sequence_length),
    tf.keras.layers.LSTM(64, return_sequences=True),
    tf.keras.layers.Dense(vocab_size, activation="softmax"),
])
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam")
print("output shape (batch, timesteps, vocab):", model.output_shape)

output shape (batch, timesteps, vocab): (None, 20, 40)

output shape (batch, timesteps, vocab): (None, 20, 40)

Every one of the 20 timesteps outputs its own 40-value probability distribution — one prediction of “what comes next” per position in the sequence.

The sampling strategy: how do you pick a token?

Given a probability distribution, there are two obvious strategies, and both are bad:

• Greedy sampling — always pick the single most likely token. This sounds safe, but it makes the model repeat itself: once it’s in a locally-likely loop ("it is a movie that is so bad that it is a""it is a movie that is so bad that it is a"), greedy sampling has no way to break out, because the most likely next word keeps leading back into the same loop.
• Uniform random sampling — ignore the model’s probabilities entirely and pick any token with equal odds. This has maximum randomness (maximum entropy) and produces word salad, because it throws away everything the model learned.

The useful strategy sits between these two extremes: sample proportionally to the model’s own probabilities. If the model says matmat has 60% probability, you pick matmat 60% of the time — but the other 40% of the time, you let something less likely (and more surprising) through. This is stochastic sampling, and greedy sampling is really just its zero-randomness special case (one token gets probability 1, everything else gets 0).

Softmax temperature: a dial between boring and chaotic

Stochastic sampling from the raw model distribution is a fixed point — it gives you no control over how random the output is. The softmax temperature fixes that: it’s a single number that reweights a distribution to be more confident (lower temperature) or more uniform (higher temperature) before you sample from it.

import numpy as np
 
def reweight_distribution(original_distribution, temperature=0.5):
    distribution = np.log(original_distribution) / temperature
    distribution = np.exp(distribution)
    return distribution / np.sum(distribution)   # renormalize to sum to 1
 
# A toy 5-word distribution: "mat" is the clear favorite at 60%
probs = np.array([0.05, 0.10, 0.60, 0.15, 0.10])
 
for temp in [0.5, 1.0, 2.0]:
    reweighted = reweight_distribution(probs, temperature=temp)
    print(f"temperature={temp}:", np.round(reweighted, 3))

import numpy as np
 
def reweight_distribution(original_distribution, temperature=0.5):
    distribution = np.log(original_distribution) / temperature
    distribution = np.exp(distribution)
    return distribution / np.sum(distribution)   # renormalize to sum to 1
 
# A toy 5-word distribution: "mat" is the clear favorite at 60%
probs = np.array([0.05, 0.10, 0.60, 0.15, 0.10])
 
for temp in [0.5, 1.0, 2.0]:
    reweighted = reweight_distribution(probs, temperature=temp)
    print(f"temperature={temp}:", np.round(reweighted, 3))

temperature=0.5: [0.006 0.025 0.889 0.056 0.025]
temperature=1.0: [0.05  0.1   0.6   0.15  0.1  ]
temperature=2.0: [0.111 0.157 0.384 0.192 0.157]

temperature=0.5: [0.006 0.025 0.889 0.056 0.025]
temperature=1.0: [0.05  0.1   0.6   0.15  0.1  ]
temperature=2.0: [0.111 0.157 0.384 0.192 0.157]

Watch what happens to the favorite word (matmat, index 2, 60% originally):

• Low temperature (0.5) squeezes almost all the probability mass onto it (89%) — the output gets more predictable, edging toward greedy sampling.
• Temperature = 1.0 leaves the distribution untouched — this is exactly the model’s own softmax output, with no reweighting at all.
• High temperature (2.0) flattens the distribution (38%, with everyone else much closer to even) — the output gets more surprising, edging toward uniform random sampling.

Low temperature narrows the distribution (lower entropy, more predictable, can get stuck in loops); high temperature flattens it (higher entropy, more creative, can degrade into nonsense at extremes). There’s no universally “correct” temperature — Géron’s book and Chollet’s own experiments both land on something around 0.5–0.8 as a practical sweet spot, but it’s always worth trying a range.

Sampling a token at a given temperature

Once you’ve reweighted the distribution, drawing an actual token is just one call to a categorical sampler — np.random.multinomialnp.random.multinomial treats the reweighted distribution as a set of dice-roll odds and returns which face came up:

import numpy as np
 
def sample_next(predictions, temperature=1.0):
    predictions = np.asarray(predictions).astype("float64")
    predictions = np.log(predictions) / temperature
    exp_preds = np.exp(predictions)
    predictions = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, predictions, 1)
    return int(np.argmax(probas))
 
vocab = ["floor", "roof", "mat", "sofa", "rug"]
probs = np.array([0.05, 0.10, 0.60, 0.15, 0.10])
 
np.random.seed(42)
picks = [sample_next(probs, temperature=1.0) for _ in range(10)]
print("10 samples at temperature=1.0:", [vocab[i] for i in picks])

import numpy as np
 
def sample_next(predictions, temperature=1.0):
    predictions = np.asarray(predictions).astype("float64")
    predictions = np.log(predictions) / temperature
    exp_preds = np.exp(predictions)
    predictions = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, predictions, 1)
    return int(np.argmax(probas))
 
vocab = ["floor", "roof", "mat", "sofa", "rug"]
probs = np.array([0.05, 0.10, 0.60, 0.15, 0.10])
 
np.random.seed(42)
picks = [sample_next(probs, temperature=1.0) for _ in range(10)]
print("10 samples at temperature=1.0:", [vocab[i] for i in picks])

10 samples at temperature=1.0: ['roof', 'mat', 'rug', 'rug', 'mat', 'mat', 'mat', 'mat', 'mat', 'mat']

10 samples at temperature=1.0: ['roof', 'mat', 'rug', 'rug', 'mat', 'mat', 'mat', 'mat', 'mat', 'mat']

matmat still wins most of the time (it has 60% probability), but roofroof and rugrug sneak in too — that occasional surprise is exactly what keeps generated text from sounding like a broken record.

Visualize it

Watch the same 6-word probability distribution reshape as the temperature slides from near-zero up to 2.0: at low temperature almost the entire bar chart collapses onto the single most likely word; as temperature rises the bars flatten out toward one another, until at high temperature every word is nearly equally likely:

Mini-checkpoint

Why does a temperature of exactly 1.01.0 leave the distribution unchanged?

• Because reweight_distributionreweight_distribution divides log(p)log(p) by the temperature before exponentiating and renormalizing. Dividing by 1.01.0 is a no-op, so exp(log(p) / 1.0)exp(log(p) / 1.0) is just pp again (up to floating-point renormalization) — temperature 1.0 is the model’s original softmax output, untouched.

🧪 Try It Yourself

Exercise 1 – Reweight a Distribution by Temperature

Exercise 2 – Compare Entropy at Low vs. High Temperature

Exercise 3 – Sample Tokens at a Chosen Temperature

Next

Continue to DeepDream — instead of generating text, run a convnet in reverse, doing gradient ascent on an image to amplify the patterns it already half-sees.