Skip to content
Python Central Hub

Regularization & Dropout

What you’ll learn

  • L1 and L2 regularization on a layer’s weights, and functools.partialfunctools.partial to avoid repeating the same arguments everywhere
  • Dropout — randomly ignoring neurons during training — and why it works
  • MC Dropout, which turns a trained Dropout model into an uncertainty estimator for free
  • max-norm regularization, which rescales weights after every training step

Why regularize at all?

Deep nets have “an incredible amount of freedom”: tens of thousands of parameters, sometimes millions. That’s exactly what lets them fit complex datasets — and exactly what lets them overfit the training set instead of learning something that generalizes. You already know one of the best fixes from Chapter 10’s toolkit: early stopping. Batch Normalization also acts as a mild regularizer as a side effect. This page covers three more dedicated techniques.

Reducing model size — the simplest regularizer

Before reaching for L1/L2 penalties or Dropout, Chollet’s suggestion is to try the cheapest fix first: make the model smaller. A model with fewer parameters has less room to simply memorize the training set, so to reduce its loss it’s forced to learn compressed, genuinely predictive patterns instead — exactly what generalizes to new data. There’s no formula for the “right” size; the workflow is to start small and add capacity only until validation loss stops improving.

Same architecture, three capacities
import tensorflow as tf
 
def build(units):
    return tf.keras.models.Sequential([
        tf.keras.layers.Dense(units, activation="relu", input_shape=(20,)),
        tf.keras.layers.Dense(units, activation="relu"),
        tf.keras.layers.Dense(1, activation="sigmoid"),
    ])
 
small_model = build(units=4)     # underfits later to overfit, but recovers slower
reference_model = build(units=16)
large_model = build(units=512)   # overfits almost immediately, noisy val loss
 
for name, model in [("small", small_model), ("reference", reference_model), ("large", large_model)]:
    print(name, "params:", model.count_params())
Same architecture, three capacities
import tensorflow as tf
 
def build(units):
    return tf.keras.models.Sequential([
        tf.keras.layers.Dense(units, activation="relu", input_shape=(20,)),
        tf.keras.layers.Dense(units, activation="relu"),
        tf.keras.layers.Dense(1, activation="sigmoid"),
    ])
 
small_model = build(units=4)     # underfits later to overfit, but recovers slower
reference_model = build(units=16)
large_model = build(units=512)   # overfits almost immediately, noisy val loss
 
for name, model in [("small", small_model), ("reference", reference_model), ("large", large_model)]:
    print(name, "params:", model.count_params())

A model that’s too small won’t overfit at all — it’ll plateau and stay there, which is itself useful information: it tells you to add capacity back before you reach for any of the regularizers below.

L1 and L2 regularization

Just like Ridge and Lasso for linear models, you can penalize a layer’s connection weights directly. L2 regularization shrinks weights toward zero smoothly; L1 pushes many of them all the way to zero, giving you a sparse model.

L2-regularized Dense layer
import tensorflow as tf
 
layer = tf.keras.layers.Dense(
    100,
    activation="elu",
    kernel_initializer="he_normal",
    kernel_regularizer=tf.keras.regularizers.l2(0.01),
)
L2-regularized Dense layer
import tensorflow as tf
 
layer = tf.keras.layers.Dense(
    100,
    activation="elu",
    kernel_initializer="he_normal",
    kernel_regularizer=tf.keras.regularizers.l2(0.01),
)

Because you’ll typically want the same activation, initializer, and regularizer on every hidden layer, functools.partialfunctools.partial saves you from repeating yourself (and from typos that only show up on the third layer):

functools.partial to avoid repetition
import tensorflow as tf
from functools import partial
 
RegularizedDense = partial(
    tf.keras.layers.Dense,
    activation="elu",
    kernel_initializer="he_normal",
    kernel_regularizer=tf.keras.regularizers.l2(0.01),
)
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    RegularizedDense(300),
    RegularizedDense(100),
    RegularizedDense(10, activation="softmax", kernel_initializer="glorot_uniform"),
])
functools.partial to avoid repetition
import tensorflow as tf
from functools import partial
 
RegularizedDense = partial(
    tf.keras.layers.Dense,
    activation="elu",
    kernel_initializer="he_normal",
    kernel_regularizer=tf.keras.regularizers.l2(0.01),
)
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    RegularizedDense(300),
    RegularizedDense(100),
    RegularizedDense(10, activation="softmax", kernel_initializer="glorot_uniform"),
])

Dropout

Dropout is one of the most popular — and most surprising — regularizers for deep nets. At every training step, every neuron (including inputs, but never outputs) has a probability pp of being temporarily ignored entirely for that step. The dropout rate pp is usually 10%–50%10%–50%.

diagram Diagram mermaid

It sounds destructive, but it works because no neuron can rely on any single neighbor being present — each one has to become independently useful, which produces a network that’s less sensitive to noise and generalizes better. Keras handles the training/prediction asymmetry (scaling outputs by the keep probability) for you automatically:

Dropout before every Dense layer
import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dropout(rate=0.2),
    tf.keras.layers.Dense(300, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.Dropout(rate=0.2),
    tf.keras.layers.Dense(100, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.Dropout(rate=0.2),
    tf.keras.layers.Dense(10, activation="softmax"),
])
Dropout before every Dense layer
import tensorflow as tf
 
model = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dropout(rate=0.2),
    tf.keras.layers.Dense(300, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.Dropout(rate=0.2),
    tf.keras.layers.Dense(100, activation="elu", kernel_initializer="he_normal"),
    tf.keras.layers.Dropout(rate=0.2),
    tf.keras.layers.Dense(10, activation="softmax"),
])

Since Dropout is only active during training, always compare training loss without dropout (measured after training) against the validation loss — with dropout on, a model can look like it isn’t overfitting even when it is.

MC Dropout

Yarin Gal and Zoubin Ghahramani’s trick: leave dropout on at prediction time and run the same input through the model many times. Because dropout is random, every pass gives a slightly different prediction — average them, and you get both a better point estimate and a genuine uncertainty measure (the standard deviation across passes), all without retraining anything.

MC Dropout: uncertainty from a trained model
import numpy as np
 
# training=True keeps Dropout active even though we are predicting
y_probas = np.stack([model(X_test_scaled, training=True) for _ in range(100)])
y_proba = y_probas.mean(axis=0)   # the MC Dropout point estimate
y_std = y_probas.std(axis=0)      # how uncertain each prediction is
MC Dropout: uncertainty from a trained model
import numpy as np
 
# training=True keeps Dropout active even though we are predicting
y_probas = np.stack([model(X_test_scaled, training=True) for _ in range(100)])
y_proba = y_probas.mean(axis=0)   # the MC Dropout point estimate
y_std = y_probas.std(axis=0)      # how uncertain each prediction is

If other layers (like BatchNormalization) also behave differently in training vs. prediction, forcing training=Truetraining=True globally isn’t safe — subclass DropoutDropout instead so only dropout is forced on:

An MCDropout layer that is always active
import tensorflow as tf
 
class MCDropout(tf.keras.layers.Dropout):
    def call(self, inputs):
        return super().call(inputs, training=True)
An MCDropout layer that is always active
import tensorflow as tf
 
class MCDropout(tf.keras.layers.Dropout):
    def call(self, inputs):
        return super().call(inputs, training=True)

Max-norm regularization

Instead of adding a penalty term to the loss, max-norm regularization rescales a neuron’s incoming weight vector after every training step, so that ‖w‖₂ ≤ r‖w‖₂ ≤ r. Smaller rr means stronger regularization, and it can also help tame unstable gradients even without Batch Normalization.

Max-norm constraint on a Dense layer
import tensorflow as tf
 
layer = tf.keras.layers.Dense(
    100,
    activation="elu",
    kernel_initializer="he_normal",
    kernel_constraint=tf.keras.constraints.max_norm(1.0),
)
Max-norm constraint on a Dense layer
import tensorflow as tf
 
layer = tf.keras.layers.Dense(
    100,
    activation="elu",
    kernel_initializer="he_normal",
    kernel_constraint=tf.keras.constraints.max_norm(1.0),
)

Visualize it

Each training step, Dropout silently switches a random subset of neurons off (shown dimmed below) — click to re-roll which ones are dropped:

sketch Dropout randomly disables neurons each step p5.js
A dense layer where a random subset of neurons blink off (dimmed) at independent, staggered training steps; click to resample every neuron at once.

🧪 Try It Yourself

Exercise 1 – L2-Regularize a Dense Layer

Exercise 2 – Add a Dropout Layer

Exercise 3 – Run MC Dropout Predictions

Next

Continue to Learning Rate Scheduling — the last lever for training deep nets faster and reaching a better final solution.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did