Skip to content
Python Central Hub

Autoencoders

What you’ll learn

  • what an autoencoder actually is, and why “just copy the input to the output” is a harder task than it sounds once you add constraints
  • how an undercomplete linear autoencoder performs PCA, and how a stacked autoencoder learns richer, non-linear codings
  • how to halve the parameter count with tied weights
  • how denoising and sparse autoencoders force useful features in different ways
  • how a convolutional autoencoder handles images properly
  • how autoencoders enable unsupervised pretraining when you have lots of unlabeled data but few labels

The idea: copy the input, but through a bottleneck

An autoencoder is always made of two parts: an encoder that compresses the input into a latent representation (the coding), and a decoder that expands that coding back out into a reconstruction that should look like the original input. Both halves are trained together, using the input itself as the target — there are no labels involved, which is why autoencoders are unsupervised (or “self-supervised”: the labels are just the inputs).

If the coding layer is smaller than the input, the autoencoder is undercomplete. It physically cannot copy the input straight through, so it’s forced to learn which features actually matter and discard the rest — much like memorizing “the even numbers from 50 down to 14” is easier than memorizing ten random numbers, because you only need to remember the pattern.

diagram Diagram mermaid

Undercomplete linear autoencoder = PCA

If every layer is linear and the loss is MSE, an undercomplete autoencoder ends up performing Principal Component Analysis — it finds the best lower-dimensional plane to project the data onto:

A linear autoencoder performing PCA
import numpy as np
import tensorflow as tf
 
# A toy 3D dataset that mostly lies on a 2D plane
rng = np.random.default_rng(42)
angles = rng.uniform(0, 2 * np.pi, 200)
X_train = np.c_[
    np.cos(angles) + rng.normal(0, 0.05, 200),
    np.sin(angles) * 0.6 + rng.normal(0, 0.05, 200),
    np.cos(angles) * 0.3 + np.sin(angles) * 0.3 + rng.normal(0, 0.05, 200),
].astype("float32")
 
encoder = tf.keras.models.Sequential([tf.keras.layers.Dense(2, input_shape=[3])])
decoder = tf.keras.models.Sequential([tf.keras.layers.Dense(3, input_shape=[2])])
autoencoder = tf.keras.models.Sequential([encoder, decoder])
 
autoencoder.compile(loss="mse", optimizer=tf.keras.optimizers.SGD(learning_rate=0.1))
history = autoencoder.fit(X_train, X_train, epochs=20, verbose=0)
 
codings = encoder.predict(X_train, verbose=0)
print("3D input shape:", X_train.shape)
print("2D codings shape:", codings.shape)
print("final training loss:", round(history.history["loss"][-1], 4))
A linear autoencoder performing PCA
import numpy as np
import tensorflow as tf
 
# A toy 3D dataset that mostly lies on a 2D plane
rng = np.random.default_rng(42)
angles = rng.uniform(0, 2 * np.pi, 200)
X_train = np.c_[
    np.cos(angles) + rng.normal(0, 0.05, 200),
    np.sin(angles) * 0.6 + rng.normal(0, 0.05, 200),
    np.cos(angles) * 0.3 + np.sin(angles) * 0.3 + rng.normal(0, 0.05, 200),
].astype("float32")
 
encoder = tf.keras.models.Sequential([tf.keras.layers.Dense(2, input_shape=[3])])
decoder = tf.keras.models.Sequential([tf.keras.layers.Dense(3, input_shape=[2])])
autoencoder = tf.keras.models.Sequential([encoder, decoder])
 
autoencoder.compile(loss="mse", optimizer=tf.keras.optimizers.SGD(learning_rate=0.1))
history = autoencoder.fit(X_train, X_train, epochs=20, verbose=0)
 
codings = encoder.predict(X_train, verbose=0)
print("3D input shape:", X_train.shape)
print("2D codings shape:", codings.shape)
print("final training loss:", round(history.history["loss"][-1], 4))

Notice the target passed to fit()fit() is X_trainX_train itself — the autoencoder’s job is to reconstruct its own input.

Stacked autoencoders

Add more hidden layers on both sides and you get a stacked (or deep) autoencoder. The architecture is typically symmetrical around the coding layer — like a sandwich. Here’s one for Fashion MNIST, with a 784 → 100 → 30 → 100 → 784 shape:

Stacked autoencoder for Fashion MNIST
import tensorflow as tf
 
(X_train_full, _), (X_test, _) = tf.keras.datasets.fashion_mnist.load_data()
X_train_full = X_train_full.astype("float32") / 255.0
X_valid, X_train = X_train_full[:5000], X_train_full[5000:]
 
stacked_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(100, activation="selu"),
    tf.keras.layers.Dense(30, activation="selu"),
])
stacked_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, activation="selu", input_shape=[30]),
    tf.keras.layers.Dense(28 * 28, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
stacked_ae = tf.keras.models.Sequential([stacked_encoder, stacked_decoder])
stacked_ae.compile(loss="binary_crossentropy", optimizer=tf.keras.optimizers.SGD(learning_rate=1.5))
 
history = stacked_ae.fit(X_train, X_train, epochs=5, validation_data=(X_valid, X_valid), verbose=0)
 
reconstructions = stacked_ae.predict(X_valid[:3], verbose=0)
print("input shape:", X_valid[:3].shape, "reconstruction shape:", reconstructions.shape)
print("final val loss:", round(history.history["val_loss"][-1], 4))
Stacked autoencoder for Fashion MNIST
import tensorflow as tf
 
(X_train_full, _), (X_test, _) = tf.keras.datasets.fashion_mnist.load_data()
X_train_full = X_train_full.astype("float32") / 255.0
X_valid, X_train = X_train_full[:5000], X_train_full[5000:]
 
stacked_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(100, activation="selu"),
    tf.keras.layers.Dense(30, activation="selu"),
])
stacked_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, activation="selu", input_shape=[30]),
    tf.keras.layers.Dense(28 * 28, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
stacked_ae = tf.keras.models.Sequential([stacked_encoder, stacked_decoder])
stacked_ae.compile(loss="binary_crossentropy", optimizer=tf.keras.optimizers.SGD(learning_rate=1.5))
 
history = stacked_ae.fit(X_train, X_train, epochs=5, validation_data=(X_valid, X_valid), verbose=0)
 
reconstructions = stacked_ae.predict(X_valid[:3], verbose=0)
print("input shape:", X_valid[:3].shape, "reconstruction shape:", reconstructions.shape)
print("final val loss:", round(history.history["val_loss"][-1], 4))

We use binary cross-entropy instead of MSE here because each pixel intensity is treated as the probability that the pixel should be “on” — framing reconstruction as a multilabel classification problem tends to converge faster than plain regression.

Tying weights

When the architecture is symmetrical, you can tie the decoder’s weights to the encoder’s — the decoder reuses the encoder’s weight matrices, transposed, and only keeps its own bias vector. This roughly halves the number of parameters and reduces overfitting:

Tying encoder/decoder weights
import tensorflow as tf
 
class DenseTranspose(tf.keras.layers.Layer):
    def __init__(self, dense, activation=None, **kwargs):
        super().__init__(**kwargs)
        self.dense = dense
        self.activation = tf.keras.activations.get(activation)
 
    def build(self, batch_input_shape):
        self.biases = self.add_weight(
            name="bias", shape=[self.dense.input_shape[-1]], initializer="zeros"
        )
        super().build(batch_input_shape)
 
    def call(self, inputs):
        z = tf.matmul(inputs, self.dense.weights[0], transpose_b=True)
        return self.activation(z + self.biases)
 
dense_1 = tf.keras.layers.Dense(100, activation="selu")
dense_2 = tf.keras.layers.Dense(30, activation="selu")
 
tied_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]), dense_1, dense_2,
])
tied_decoder = tf.keras.models.Sequential([
    DenseTranspose(dense_2, activation="selu"),
    DenseTranspose(dense_1, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
tied_ae = tf.keras.models.Sequential([tied_encoder, tied_decoder])
print("tied AE trainable params:", tied_ae.count_params())
Tying encoder/decoder weights
import tensorflow as tf
 
class DenseTranspose(tf.keras.layers.Layer):
    def __init__(self, dense, activation=None, **kwargs):
        super().__init__(**kwargs)
        self.dense = dense
        self.activation = tf.keras.activations.get(activation)
 
    def build(self, batch_input_shape):
        self.biases = self.add_weight(
            name="bias", shape=[self.dense.input_shape[-1]], initializer="zeros"
        )
        super().build(batch_input_shape)
 
    def call(self, inputs):
        z = tf.matmul(inputs, self.dense.weights[0], transpose_b=True)
        return self.activation(z + self.biases)
 
dense_1 = tf.keras.layers.Dense(100, activation="selu")
dense_2 = tf.keras.layers.Dense(30, activation="selu")
 
tied_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]), dense_1, dense_2,
])
tied_decoder = tf.keras.models.Sequential([
    DenseTranspose(dense_2, activation="selu"),
    DenseTranspose(dense_1, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
tied_ae = tf.keras.models.Sequential([tied_encoder, tied_decoder])
print("tied AE trainable params:", tied_ae.count_params())

Denoising and sparse autoencoders

Denoising autoencoders add noise (Gaussian noise, or randomly zeroed-out inputs via DropoutDropout) to the input, then train the network to recover the clean version. It cannot cheat by copying — it has to learn what the data actually looks like:

Denoising autoencoder with Dropout
import tensorflow as tf
 
dropout_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(100, activation="selu"),
    tf.keras.layers.Dense(30, activation="selu"),
])
dropout_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, activation="selu", input_shape=[30]),
    tf.keras.layers.Dense(28 * 28, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
dropout_ae = tf.keras.models.Sequential([dropout_encoder, dropout_decoder])
print("layers with Dropout active only during training:",
      [type(l).__name__ for l in dropout_encoder.layers])
Denoising autoencoder with Dropout
import tensorflow as tf
 
dropout_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(100, activation="selu"),
    tf.keras.layers.Dense(30, activation="selu"),
])
dropout_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, activation="selu", input_shape=[30]),
    tf.keras.layers.Dense(28 * 28, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
dropout_ae = tf.keras.models.Sequential([dropout_encoder, dropout_decoder])
print("layers with Dropout active only during training:",
      [type(l).__name__ for l in dropout_encoder.layers])

Sparse autoencoders take the opposite approach: keep the coding layer large, but penalize the model unless most of its neurons stay near zero. This pushes each active neuron to represent something genuinely useful:

Sparse autoencoder with an activity penalty
import tensorflow as tf
 
sparse_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(100, activation="selu"),
    tf.keras.layers.Dense(300, activation="sigmoid"),
    tf.keras.layers.ActivityRegularization(l1=1e-3),  # penalizes non-zero codings
])
sparse_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, activation="selu", input_shape=[300]),
    tf.keras.layers.Dense(28 * 28, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
sparse_ae = tf.keras.models.Sequential([sparse_encoder, sparse_decoder])
print("coding layer size:", sparse_encoder.layers[-2].units)
Sparse autoencoder with an activity penalty
import tensorflow as tf
 
sparse_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=[28, 28]),
    tf.keras.layers.Dense(100, activation="selu"),
    tf.keras.layers.Dense(300, activation="sigmoid"),
    tf.keras.layers.ActivityRegularization(l1=1e-3),  # penalizes non-zero codings
])
sparse_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, activation="selu", input_shape=[300]),
    tf.keras.layers.Dense(28 * 28, activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
sparse_ae = tf.keras.models.Sequential([sparse_encoder, sparse_decoder])
print("coding layer size:", sparse_encoder.layers[-2].units)

Convolutional autoencoders

Dense layers ignore spatial structure, so for real images a convolutional autoencoder works far better: the encoder downsamples with strided convolutions, and the decoder upsamples back with transpose convolutions:

Convolutional autoencoder for Fashion MNIST
import tensorflow as tf
 
conv_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Reshape([28, 28, 1], input_shape=[28, 28]),
    tf.keras.layers.Conv2D(16, kernel_size=3, padding="same", activation="selu"),
    tf.keras.layers.MaxPool2D(pool_size=2),
    tf.keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="selu"),
    tf.keras.layers.MaxPool2D(pool_size=2),
    tf.keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="selu"),
    tf.keras.layers.MaxPool2D(pool_size=2),
])
conv_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Conv2DTranspose(32, kernel_size=3, strides=2, padding="valid",
                                     activation="selu", input_shape=[3, 3, 64]),
    tf.keras.layers.Conv2DTranspose(16, kernel_size=3, strides=2, padding="same", activation="selu"),
    tf.keras.layers.Conv2DTranspose(1, kernel_size=3, strides=2, padding="same", activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
conv_ae = tf.keras.models.Sequential([conv_encoder, conv_decoder])
print("bottleneck shape:", conv_encoder.output_shape)
Convolutional autoencoder for Fashion MNIST
import tensorflow as tf
 
conv_encoder = tf.keras.models.Sequential([
    tf.keras.layers.Reshape([28, 28, 1], input_shape=[28, 28]),
    tf.keras.layers.Conv2D(16, kernel_size=3, padding="same", activation="selu"),
    tf.keras.layers.MaxPool2D(pool_size=2),
    tf.keras.layers.Conv2D(32, kernel_size=3, padding="same", activation="selu"),
    tf.keras.layers.MaxPool2D(pool_size=2),
    tf.keras.layers.Conv2D(64, kernel_size=3, padding="same", activation="selu"),
    tf.keras.layers.MaxPool2D(pool_size=2),
])
conv_decoder = tf.keras.models.Sequential([
    tf.keras.layers.Conv2DTranspose(32, kernel_size=3, strides=2, padding="valid",
                                     activation="selu", input_shape=[3, 3, 64]),
    tf.keras.layers.Conv2DTranspose(16, kernel_size=3, strides=2, padding="same", activation="selu"),
    tf.keras.layers.Conv2DTranspose(1, kernel_size=3, strides=2, padding="same", activation="sigmoid"),
    tf.keras.layers.Reshape([28, 28]),
])
conv_ae = tf.keras.models.Sequential([conv_encoder, conv_decoder])
print("bottleneck shape:", conv_encoder.output_shape)

Unsupervised pretraining

If you have a huge pile of unlabeled data but only a few thousand labeled examples, train a stacked autoencoder on all of it first, then reuse the encoder’s layers as the base of your real classifier — optionally freezing them — and fine-tune on your small labeled set. The autoencoder never sees a single label; it just learns generally useful feature detectors that your classifier can build on top of.

Visualize it

Watch a bottleneck squeeze a wide input signal down to just a few latent values, then expand it back out. As “training” progresses, the reconstruction (right) gradually converges onto the original input (left) — that’s the bottleneck being forced to learn which features actually matter:

sketch Autoencoder: compress through a bottleneck, then reconstruct p5.js
An input signal is compressed to 3 latent values in the bottleneck, then decoded back out. The reconstruction slowly converges to the input as training progresses.

Mini-checkpoint

If an autoencoder reconstructs its training data perfectly, is that necessarily a good sign?

  • Not always — an overly powerful autoencoder can learn to map each input to an arbitrary unique code (basically memorizing an index), reconstruct perfectly, and still have learned nothing useful about the data’s structure.

🧪 Try It Yourself

Exercise 1 – Build an Undercomplete Autoencoder

Exercise 2 – Train on the Input Itself

Exercise 3 – Add a Denoising Dropout Layer

Next

Continue to Variational Autoencoders (VAEs) — turn that latent code into a proper probability distribution you can sample from to generate brand-new data.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did