Every layer you’ve built so far ends with activation="relu"activation="relu" or "softmax""softmax" — but what are those, and why do they matter so much? This page is about the small non-linear function every neuron applies to its weighted sum, and why skipping it would quietly turn your deep network back into a single linear model.

Why activations are needed

Without activations, layers collapse into a single linear transformation. Chain two linear functions together — say f(x) = 2x + 3f(x) = 2x + 3 and g(x) = 5x - 1g(x) = 5x - 1 — and you just get another linear function: f(g(x)) = 10x + 1f(g(x)) = 10x + 1. No matter how many layers you stack, a network without non-linear activations is mathematically equivalent to a single layer. A large-enough network with non-linear activations, on the other hand, can approximate essentially any continuous function.

There’s a second, more practical reason activations changed over time. The original perceptron used a hard step function, which is flat everywhere except at zero — so its gradient is zero almost everywhere, and Gradient Descent has nothing to push against. Backpropagation needed a function with a well-defined, non-zero derivative everywhere. That’s why the first successful MLPs replaced the step function with the smooth logistic (sigmoid) function.

ReLU

ReLU(x) = max(0, x)ReLU(x) = max(0, x)

Common choice for hidden layers.

Pros:

• simple and fast to compute
• helps with vanishing gradient (compared to sigmoid), since it doesn’t saturate for positive inputs

Cons:

• not differentiable at x = 0x = 0 (in practice, this rarely causes problems)
• “dead” neurons: if a neuron’s weighted sum is always negative, its gradient is always zero and it stops learning

Sigmoid

sigmoid(x) = 1 / (1 + e^-x)sigmoid(x) = 1 / (1 + e^-x)

Maps any real number to (0, 1), so its output can be read as a probability.

Used for:

• binary classification output (probability of the positive class)

Tanh

tanh(x) = 2·sigmoid(2x) − 1tanh(x) = 2·sigmoid(2x) − 1

Same S-shape as sigmoid, but squashes to (−1, 1) instead of (0, 1). Because its output tends to be centered around 0, it often helps hidden layers converge a little faster than sigmoid does.

Softmax

Turns a vector of raw scores (logits) into a probability distribution over classes — every value lands in (0, 1) and they all sum to 1:

softmax(z)_i = exp(z_i) / Σ_j exp(z_j)softmax(z)_i = exp(z_i) / Σ_j exp(z_j)

Used for:

• multiclass classification output, when each instance belongs to exactly one class out of several

  flowchart LR
  Z[Logits] --> S[Softmax] --> P[Class probabilities]

Typical choices

• hidden layers: ReLU
• binary output: sigmoid
• multiclass output: softmax

Picking the output layer for your actual problem

The three worked examples later in this phase (IMDB, Reuters, house prices) each end in a different output layer — here’s the full picture of why, one problem type at a time:

• Regression (predict any real number, like a house price) — usually no activation at all, so the output is free to be any value. If you know the target must be positive, ReLU or its smooth cousin softplus(z) = log(1 + exp(z))softplus(z) = log(1 + exp(z)) works too. If the target is naturally bounded, sigmoid (0 to 1) or tanh (−1 to 1) can work — but then you must scale your labels into that same range first.
• Binary classification (2 mutually exclusive classes, like IMDB) — one output neuron with sigmoid. Its single number is P(positive class)P(positive class).
• Multiclass, single-label classification (exactly one class out of several, like Reuters’ 46 topics) — one output neuron per class, with softmax over the whole layer, so the outputs form one probability distribution that sums to 1.
• Multilabel, multiclass classification (an example can belong to several classes at once — e.g. tagging an email as both “spam” and “urgent”) — one output neuron per label, each with its own sigmoid, not softmax. Every neuron answers an independent yes/no question, so the outputs do not need to sum to 1:

from tensorflow import keras
 
model = keras.Sequential([
    keras.layers.Dense(16, activation="relu", input_shape=(10,)),
    keras.layers.Dense(2, activation="sigmoid"),  # 2 labels, each independent
])
model.compile(loss="binary_crossentropy", optimizer="rmsprop", metrics=["accuracy"])

from tensorflow import keras
 
model = keras.Sequential([
    keras.layers.Dense(16, activation="relu", input_shape=(10,)),
    keras.layers.Dense(2, activation="sigmoid"),  # 2 labels, each independent
])
model.compile(loss="binary_crossentropy", optimizer="rmsprop", metrics=["accuracy"])

Using softmaxsoftmax here would be a bug: it would force P(spam) + P(urgent) = 1P(spam) + P(urgent) = 1, as if an email couldn’t be both at once.

Activations in tf.keras

You already choose an activation every time you build a DenseDense layer. Here’s how the three from this page look side by side, plus a peek at what each one actually outputs on a batch of raw scores:

import numpy as np
from tensorflow import keras
 
logits = np.array([[2.0, -1.0, 0.5]])
 
relu = keras.activations.relu(logits)
sigmoid = keras.activations.sigmoid(logits)
softmax = keras.activations.softmax(logits)
 
print("ReLU:   ", relu.numpy())
print("Sigmoid:", sigmoid.numpy())
print("Softmax:", softmax.numpy(), "-> sums to", softmax.numpy().sum())

import numpy as np
from tensorflow import keras
 
logits = np.array([[2.0, -1.0, 0.5]])
 
relu = keras.activations.relu(logits)
sigmoid = keras.activations.sigmoid(logits)
softmax = keras.activations.softmax(logits)
 
print("ReLU:   ", relu.numpy())
print("Sigmoid:", sigmoid.numpy())
print("Softmax:", softmax.numpy(), "-> sums to", softmax.numpy().sum())

from tensorflow import keras
 
model = keras.Sequential([
    keras.layers.Dense(300, activation="relu", input_shape=(784,)),  # hidden: ReLU
    keras.layers.Dense(100, activation="relu"),                      # hidden: ReLU
    keras.layers.Dense(10, activation="softmax"),                    # output: softmax
])
model.compile(loss="sparse_categorical_crossentropy",
              optimizer="sgd",
              metrics=["accuracy"])

from tensorflow import keras
 
model = keras.Sequential([
    keras.layers.Dense(300, activation="relu", input_shape=(784,)),  # hidden: ReLU
    keras.layers.Dense(100, activation="relu"),                      # hidden: ReLU
    keras.layers.Dense(10, activation="softmax"),                    # output: softmax
])
model.compile(loss="sparse_categorical_crossentropy",
              optimizer="sgd",
              metrics=["accuracy"])

Visualize it

Each activation reshapes the neuron’s signal differently: ReLU zeroes out negatives and passes positives straight through, sigmoid squashes everything into 0–1, and tanh squashes into −1–1. Their shapes are why they behave so differently:

Mini-checkpoint

If your model is predicting 10 classes, what activation is typical in the last layer?

(Softmax.)

Next

You now know the perceptron, the MLP, and the activations that make them work. Continue to Building Neural Networks with Keras (Sequential and Functional API) to see the different ways to assemble these pieces into an actual tf.kerastf.keras model — before moving on to Phase 2’s training toolkit.

🧪 Try It Yourself

Exercise 1 – Apply ReLU

Exercise 2 – A Sigmoid Output Layer

Exercise 3 – Softmax Sums to 1