Skip to content
Python Central Hub

Learning Rate Scheduling

What you’ll learn

  • why a single constant learning rate is always a compromise
  • power scheduling, exponential scheduling, and piecewise constant scheduling
  • performance scheduling with ReduceLROnPlateauReduceLROnPlateau
  • the 1cycle schedule, and why Géron calls it out as the strongest option

Why schedule the learning rate at all?

Too high, and training diverges. Too low, and it converges but takes forever. Slightly too high, and it makes fast progress early on, then dances around the optimum without ever quite settling. A learning rate schedule sidesteps the whole trade-off: start large (fast early progress) and shrink it once progress slows down, reaching a better solution faster than any single constant rate could.

diagram Diagram mermaid

Get the base rate right before you schedule it

A schedule only decays a starting point — if that starting point is badly wrong, no schedule will save the run. Chollet’s diagnostic for a training loss that simply won’t move: don’t reach for a fancier optimizer first, tune the learning rate and batch size, since they’re usually enough on their own to get things moving. A learning rate of 1.01.0 on RMSprop can make training overshoot so badly that accuracy gets stuck around 30–40%; dropping it to 1e-21e-2 on the exact same model and data can be the entire fix:

Same model, same data -- only the learning rate changes
import numpy as np
import tensorflow as tf
 
np.random.seed(42)
X = np.random.rand(200, 20).astype("float32")
y = (X.sum(axis=1) > 10).astype("float32")
 
def train(learning_rate, epochs=5):
    model = tf.keras.models.Sequential([
        tf.keras.layers.Dense(32, activation="relu", input_shape=(20,)),
        tf.keras.layers.Dense(1, activation="sigmoid"),
    ])
    model.compile(
        optimizer=tf.keras.optimizers.RMSprop(learning_rate),
        loss="binary_crossentropy",
        metrics=["accuracy"],
    )
    history = model.fit(X, y, epochs=epochs, batch_size=32, verbose=0)
    return history.history["accuracy"][-1]
 
print("lr=1.0  final accuracy:", round(train(1.0), 2))
print("lr=1e-2 final accuracy:", round(train(1e-2), 2))
Same model, same data -- only the learning rate changes
import numpy as np
import tensorflow as tf
 
np.random.seed(42)
X = np.random.rand(200, 20).astype("float32")
y = (X.sum(axis=1) > 10).astype("float32")
 
def train(learning_rate, epochs=5):
    model = tf.keras.models.Sequential([
        tf.keras.layers.Dense(32, activation="relu", input_shape=(20,)),
        tf.keras.layers.Dense(1, activation="sigmoid"),
    ])
    model.compile(
        optimizer=tf.keras.optimizers.RMSprop(learning_rate),
        loss="binary_crossentropy",
        metrics=["accuracy"],
    )
    history = model.fit(X, y, epochs=epochs, batch_size=32, verbose=0)
    return history.history["accuracy"][-1]
 
print("lr=1.0  final accuracy:", round(train(1.0), 2))
print("lr=1e-2 final accuracy:", round(train(1e-2), 2))

Once training actually gets started at a sane base rate, that’s when scheduling (below) starts paying off — squeezing out a better final result faster than a constant rate ever could.

Power scheduling

The learning rate becomes a function of the iteration number tt:

η(t) = η₀ / (1 + t/s)^cη(t) = η₀ / (1 + t/s)^c

It drops quickly at first, then more and more slowly. In Keras this is the easiest schedule to wire up — just set decaydecay on the optimizer (Keras assumes c = 1c = 1, and decaydecay is 1/s1/s):

Power scheduling via optimizer decay
import tensorflow as tf
 
optimizer = tf.keras.optimizers.legacy.SGD(learning_rate=0.01, decay=1e-4)
Power scheduling via optimizer decay
import tensorflow as tf
 
optimizer = tf.keras.optimizers.legacy.SGD(learning_rate=0.01, decay=1e-4)

Exponential scheduling

η(t) = η₀ · 0.1^(t/s)η(t) = η₀ · 0.1^(t/s) — the learning rate is slashed by a factor of 10 every ss steps, a steadier decay than power scheduling’s fast-then-slow curve. Define a schedule function and hand it to a LearningRateSchedulerLearningRateScheduler callback:

Exponential scheduling with a callback
import tensorflow as tf
 
def exponential_decay(lr0, s):
    def exponential_decay_fn(epoch):
        return lr0 * 0.1 ** (epoch / s)
    return exponential_decay_fn
 
exponential_decay_fn = exponential_decay(lr0=0.01, s=20)
lr_scheduler = tf.keras.callbacks.LearningRateScheduler(exponential_decay_fn)
 
# history = model.fit(X_train, y_train, epochs=20, callbacks=[lr_scheduler])
Exponential scheduling with a callback
import tensorflow as tf
 
def exponential_decay(lr0, s):
    def exponential_decay_fn(epoch):
        return lr0 * 0.1 ** (epoch / s)
    return exponential_decay_fn
 
exponential_decay_fn = exponential_decay(lr0=0.01, s=20)
lr_scheduler = tf.keras.callbacks.LearningRateScheduler(exponential_decay_fn)
 
# history = model.fit(X_train, y_train, epochs=20, callbacks=[lr_scheduler])

Piecewise constant scheduling

Use one learning rate for a stretch of epochs, then drop to a smaller one, and so on. Simple, but it takes some trial and error to pick the right rates and how long to hold each one:

Piecewise constant scheduling
import tensorflow as tf
 
def piecewise_constant_fn(epoch):
    if epoch < 5:
        return 0.01
    elif epoch < 15:
        return 0.005
    else:
        return 0.001
 
lr_scheduler = tf.keras.callbacks.LearningRateScheduler(piecewise_constant_fn)
Piecewise constant scheduling
import tensorflow as tf
 
def piecewise_constant_fn(epoch):
    if epoch < 5:
        return 0.01
    elif epoch < 15:
        return 0.005
    else:
        return 0.001
 
lr_scheduler = tf.keras.callbacks.LearningRateScheduler(piecewise_constant_fn)

Performance scheduling

Instead of a fixed formula, watch the validation loss (the same signal early stopping uses) and cut the learning rate whenever it stops improving:

ReduceLROnPlateau
import tensorflow as tf
 
lr_scheduler = tf.keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)
 
# multiplies the learning rate by 0.5 whenever val_loss hasn't improved for 5 epochs
ReduceLROnPlateau
import tensorflow as tf
 
lr_scheduler = tf.keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)
 
# multiplies the learning rate by 0.5 whenever val_loss hasn't improved for 5 epochs

The tf.keras.optimizers.schedules approach

tf.kerastf.keras also lets you attach a schedule object directly to the optimizer, which updates the learning rate every step (not just every epoch) and gets saved along with the model:

ExponentialDecay schedule object
import tensorflow as tf
 
batch_size = 32
s = 20 * len(X_train) // batch_size  # steps for ~20 epochs
 
learning_rate = tf.keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)
optimizer = tf.keras.optimizers.SGD(learning_rate)
ExponentialDecay schedule object
import tensorflow as tf
 
batch_size = 32
s = 20 * len(X_train) // batch_size  # steps for ~20 epochs
 
learning_rate = tf.keras.optimizers.schedules.ExponentialDecay(0.01, s, 0.1)
optimizer = tf.keras.optimizers.SGD(learning_rate)

1cycle scheduling

Leslie Smith’s 1cycle schedule (2018) does something different from every schedule above: it increases the learning rate from η₀η₀ up to a peak η₁η₁ during the first half of training, then decreases it back down to η₀η₀ during the second half, finishing the last few epochs by dropping several orders of magnitude further. Momentum is often varied in the opposite direction — starting high, dipping in the middle, then climbing back up. On CIFAR10, Smith reported reaching 91.9% validation accuracy in just 100 epochs with 1cycle, versus 90.3% in 800 epochs the standard way — the same architecture, radically less training.

A minimal 1cycle-style callback
import tensorflow as tf
 
class OneCycleScheduler(tf.keras.callbacks.Callback):
    def __init__(self, iterations, max_lr=1e-3, start_lr=None, last_iterations=None, last_lr=None):
        self.iterations = iterations
        self.max_lr = max_lr
        self.start_lr = start_lr or max_lr / 10
        self.last_iterations = last_iterations or iterations // 10 + 1
        self.half_iteration = (iterations - self.last_iterations) // 2
        self.last_lr = last_lr or self.start_lr / 1000
        self.iteration = 0
 
    def _interpolate(self, iter1, iter2, lr1, lr2):
        return (lr2 - lr1) * (self.iteration - iter1) / (iter2 - iter1) + lr1
 
    def on_batch_begin(self, batch, logs=None):
        if self.iteration < self.half_iteration:
            lr = self._interpolate(0, self.half_iteration, self.start_lr, self.max_lr)
        elif self.iteration < 2 * self.half_iteration:
            lr = self._interpolate(self.half_iteration, 2 * self.half_iteration, self.max_lr, self.start_lr)
        else:
            lr = self._interpolate(2 * self.half_iteration, self.iterations, self.start_lr, self.last_lr)
        self.iteration += 1
        self.model.optimizer.learning_rate = lr
A minimal 1cycle-style callback
import tensorflow as tf
 
class OneCycleScheduler(tf.keras.callbacks.Callback):
    def __init__(self, iterations, max_lr=1e-3, start_lr=None, last_iterations=None, last_lr=None):
        self.iterations = iterations
        self.max_lr = max_lr
        self.start_lr = start_lr or max_lr / 10
        self.last_iterations = last_iterations or iterations // 10 + 1
        self.half_iteration = (iterations - self.last_iterations) // 2
        self.last_lr = last_lr or self.start_lr / 1000
        self.iteration = 0
 
    def _interpolate(self, iter1, iter2, lr1, lr2):
        return (lr2 - lr1) * (self.iteration - iter1) / (iter2 - iter1) + lr1
 
    def on_batch_begin(self, batch, logs=None):
        if self.iteration < self.half_iteration:
            lr = self._interpolate(0, self.half_iteration, self.start_lr, self.max_lr)
        elif self.iteration < 2 * self.half_iteration:
            lr = self._interpolate(self.half_iteration, 2 * self.half_iteration, self.max_lr, self.start_lr)
        else:
            lr = self._interpolate(2 * self.half_iteration, self.iterations, self.start_lr, self.last_lr)
        self.iteration += 1
        self.model.optimizer.learning_rate = lr

Visualize it

Each schedule decays the learning rate very differently. Power scheduling drops fast then flattens out; exponential scheduling keeps slashing it by the same factor; 1cycle rises first, then falls, then collapses at the very end:

sketch Learning rate schedules compared p5.js
Power, exponential, and 1cycle schedules plotted over the course of training, with a marker tracing each one from epoch 0 to epoch T.

Mini-checkpoint

You’ve trained for a while and validation loss has plateaued for the last several epochs, but you don’t want to hand-pick a decay formula. Which callback fits best?

  • ReduceLROnPlateauReduceLROnPlateau — it watches validation loss directly and cuts the learning rate only when progress actually stalls.

🧪 Try It Yourself

Exercise 1 – Build an Exponential Decay Schedule

Exercise 2 – ReduceLROnPlateau Callback

Exercise 3 – A Piecewise Constant Schedule Function

Next

Continue to The Universal Workflow of Machine Learning — you now have a full toolkit for training deep nets; the next three pages zoom back out to the project-level workflow that toolkit fits into, before Phase 3 puts it all to work on real images.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did