The Same Network in PyTorch

Every other page in this module is Keras. That is a deliberate choice — learning one framework properly beats learning two badly — but it leaves an obvious question unanswered: how much of what you have learned is about neural networks, and how much is about TensorFlow?

This page answers it by measurement. The same architecture is built in both frameworks, given byte-identical weights, and run on the same batch. Whatever differs is framework; whatever matches is the subject.

What you’ll learn

The line-by-line translation between keras.Sequential and torch.nn.Sequential.
What fit() hides, written out as the loop PyTorch makes you write yourself.
The measured agreement: gradients matching to 7.45e-09 on gradients of size 0.0386.
Identical training producing identical accuracy, and where the wall clock differs.
The differences that are real and easy to miss: initialisers, epsilon, and bias.

The same model, twice

python

# Keras
model = keras.Sequential([
    keras.layers.Input((784,)),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dense(10),
])
 
# PyTorch
model = nn.Sequential(
    nn.Linear(784, 128),
    nn.ReLU(),
    nn.Linear(128, 10),
)

Both report 101,770 parameters. Three differences are visible immediately and all three are notation:

Activations are layers in PyTorch. Keras folds relu into Dense as an argument; nn.Sequential needs nn.ReLU() as its own entry. There is no computational difference — Keras is applying exactly the same function after the matrix multiply.
PyTorch needs no Input. Shapes are inferred when data first flows through, so nn.Linear(784, 128) states its input size directly rather than deriving it.
Weight matrices are transposed. Keras stores a Dense kernel as (in, out); torch.nn.Linear stores it as (out, in). Copying weights between them requires a .T, and forgetting it is the single most common way this comparison goes wrong — the shapes happen to be compatible when in == out, so it fails silently on square layers.

Do they compute the same thing?

Give both models identical weights, push the same 256-row batch through, take the loss and one backward pass:

Horizontal bars on a log axis showing the absolute difference between Keras and PyTorch for six quantities: forward logits at 4.17e-07, loss at 2.38e-07, and four gradient tensors between 7.45e-09 and 1.86e-08. A dashed line marks 1e-6, and every bar is to the left of it. — Every difference is at or below 4.17e-07 while the gradients themselves reach 0.0386 — so the relative disagreement is around 1e-5 at worst and 1e-7 typically. That is float32 rounding, not a difference in the mathematics. The forward pass disagrees more than the gradients do, which is the ordering you would expect: the logits are the largest numbers in the computation, so they carry the most absolute rounding error.

Quantity	Largest absolute difference
Forward logits	4.17e-07
Loss	2.38e-07
Gradient: layer 1 weight	7.45e-09
Gradient: layer 1 bias	9.31e-09
Gradient: layer 2 weight	1.86e-08
Gradient: layer 2 bias	8.38e-09

Read those against the scale of the thing being measured. The gradients reach 0.0386, so a disagreement of 7.45e-09 is roughly one part in five million — the two libraries are running the same arithmetic in a different order and float32 is rounding differently.

This is the page’s main result. Backpropagation, cross-entropy, ReLU and dense layers are not TensorFlow features. Everything Phase 1 taught transfers unchanged.

What `fit()` was doing

The frameworks diverge sharply in what they hand you. Keras gives you fit(). PyTorch gives you the loop:

python

# What Keras does for you, written out
optimiser = torch.optim.SGD(model.parameters(), lr=0.1)
for epoch in range(EPOCHS):
    for start in range(0, len(x), BATCH):
        batch_x = torch.tensor(x[start:start + BATCH])
        batch_y = torch.tensor(y[start:start + BATCH])
        optimiser.zero_grad()                                   # 1
        loss = F.cross_entropy(model(batch_x), batch_y)         # 2
        loss.backward()                                         # 3
        optimiser.step()                                        # 4

Four lines, and each one corresponds to something fit() does silently:

zero_grad() — PyTorch accumulates gradients into .grad, so they must be cleared. This is the same += accumulation the autograd page builds by hand, and it is deliberate: it lets you sum gradients across several backward passes. Forget it and every step uses the sum of all previous gradients.
The forward pass and loss, identical in substance to Keras’s compile(loss=...).
backward() — walks the recorded graph. GradientTape records the same graph; the difference is that a tape is opened explicitly and PyTorch records always.
step() — applies the update. Keras’s apply_gradients, spelled shorter.

Whether that is a cost or a benefit depends entirely on what you are doing, and the honest answer is that Phase 8’s custom training loops page has you writing the same four steps in TensorFlow the moment you need control.

diagram Diagram mermaid

  flowchart TD
  A["one architecture"] --> K["keras.Sequential
101,770 parameters"]
  A --> T["torch.nn.Sequential
101,770 parameters"]
  K --> W["identical weights
copied across, with a transpose"]
  T --> W
  W --> G["gradients agree to 7.45e-09
on gradients of size 0.0386"]
  G --> R["same maths"]
  K --> D["different defaults:
init spread, Adam epsilon, bias"]
  T --> D
  D --> S["different results
if you never set them"]

Training them side by side

Same initial weights, same batch order, same optimiser and learning rate, eight epochs, three seeds:

figure MNIST, 12,000 rows, SGD at 0.1, 8 epochs, 3 seeds matplotlib

The loss curves are indistinguishable and the final losses differ by 0.000184. Test accuracy is 0.8990 for both, to four decimal places, with the same 0.0040 spread across seeds — the seeds move both frameworks identically because they start from the same weights. The only column that separates them is wall clock: 2.1 s against 3.9 s, and that gap is about framework overhead on a small model, not about arithmetic.

	Keras	PyTorch
Final training loss	0.2204	0.2203
Test accuracy	0.8990	0.8990
Spread across 3 seeds	0.0040	0.0040
Seconds for 8 epochs	3.9	2.1

The accuracy gap is 0.0000. Do not over-read the timing: this is a 101,770-parameter model on CPU, where per-step framework overhead dominates and PyTorch’s eager loop has less of it than fit()’s callback and metric machinery. On a large model on a GPU the comparison is different, and neither of those is available here to test it.

The differences that actually bite

The mathematics matches. The defaults do not, and they are the reason two “identical” models can train differently when nobody has made a mistake.

Left: bars of the standard deviation of initial weights for a 784-to-128 layer, 0.04687 for Keras with glorot_uniform against 0.02061 for torch with kaiming_uniform. Right: grouped bars on a log axis of Adam defaults — learning rate 0.001 for both, epsilon 1e-07 against 1e-08, beta_1 0.9 for both. — Same layer, same shape, weights initialised 2.3x apart: Keras uses glorot_uniform at sd 0.04687, torch uses kaiming_uniform(a=sqrt(5)) at sd 0.02061. Adam agrees on the learning rate and the betas but not on epsilon — 1e-7 against 1e-8, a factor of ten in the term that stops division by zero. Neither is wrong; they are different conventions, and a model that trains in one framework and not the other is usually meeting one of them.

Default	Keras	PyTorch
Dense/Linear initialiser	`glorot_uniform`	`kaiming_uniform(a=√5)`
Initial weight sd (784 → 128)	0.04687	0.02061
Initial bias	all zeros	uniform, up to 0.03538
Adam learning rate	0.001	0.001
Adam `beta_1`	0.9	0.9
Adam `epsilon`	1e-07	1e-08

Three things to take from that table.

The initialisers differ by 2.3×. Keras’s Glorot scales by fan-in and fan-out; PyTorch’s Kaiming variant scales by fan-in alone with a gain fixed for a leaky-ReLU slope of √5 — a default that is famously not what most people would choose deliberately. Since Phase 2 measures how much initial scale matters to deep networks, this is not cosmetic.

PyTorch initialises biases randomly; Keras zeros them. A small difference on one layer, and one more reason two fresh models diverge.

Adam’s epsilon differs by 10×. It rarely matters — until you are training in float16, where 1e-8 underflows and 1e-7 does not. Phase 8’s mixed precision page is where that stops being trivia.

sketch The same layer, two defaults p5.js

Click a framework. The histogram is the measured spread of initial weights for a 784 to 128 layer under each library's default initialiser.

Which one should you use?

Not a question this page can settle with a measurement, so here is the honest version.

The maths is the same, to 7.45e-09. Nothing you learn about networks in one is wasted in the other.

Keras is faster to write for anything that fits the fit() shape, and most supervised training does. PyTorch’s loop is four lines you will write correctly the second time and forget zero_grad() in on the first.

PyTorch dominates research code, so papers, reference implementations and pretrained checkpoints are disproportionately in it. That is an ecosystem fact rather than a technical one, and it is the strongest practical argument.

The deployment stories differ. Phase 8’s SavedModel, TensorFlow Lite and TF Serving pages have direct PyTorch equivalents (TorchScript, ExecuTorch, TorchServe) with the same shape and different names.

sketch The measured table, ranked p5.js

Click a column to rank every row by it. The bars are that column's values and the highest and lowest are computed from the numbers, not written in.

Pitfalls

Copying weights without transposing. (in, out) against (out, in). Silent on square layers.
Forgetting zero_grad(). Gradients accumulate by design; the model trains on the running sum and diverges in a way that looks like a bad learning rate.
Assuming identical defaults. The initialiser differs by 2.3× and Adam’s epsilon by 10×, before you have written a line of your own.
Reading the wall clock as a framework ranking. 2.1 s against 3.9 s is per-step overhead on a small CPU model, measured with no GPU present.
Calling .numpy() on a tensor that requires grad. PyTorch raises; use .detach().numpy(), which is the explicit version of the same thing GradientTape handles by scoping.
Comparing model.eval() and training=False casually. Both switch dropout and batch-norm behaviour, and both are easy to forget — the resulting metric is wrong but entirely plausible.

Recap

The same architecture is 101,770 parameters in both frameworks.
With identical weights, gradients agree to 7.45e-09 against gradients of size 0.0386 — roughly one part in five million, which is float32 rounding.
Trained identically, both reach 0.8990 test accuracy with the same 0.0040 seed spread; final losses differ by 0.000184.
Wall clock differed: 2.1 s against 3.9 s for 8 epochs, which is per-step overhead on a small CPU model rather than a general ranking.
fit() is the four-line loop — zero_grad, forward, backward, step — and PyTorch makes you write it.
The real differences are defaults: initial weight sd 0.04687 against 0.02061, zeroed biases against random ones, and Adam’s epsilon at 1e-7 against 1e-8.

Back to the first complete project, in Keras: First Example — Classifying Movie Reviews (IMDB).

pch.quizTag pch.quizDefaultTitle

With identical weights, Keras and PyTorch gradients differed by at most 7.45e-09 while the gradients themselves reached 0.0386. What does that establish?

pch.quizShowAnswer

B — The two libraries implement the same mathematics and differ only in float32 rounding, so everything you learn about networks transfers between them
Why must PyTorch code call `optimiser.zero_grad()` before every backward pass?

pch.quizShowAnswer

B — Because gradients accumulate into `.grad` by design — without clearing, each step uses the sum of all previous gradients
A `Dense` kernel is stored as (in, out) and `nn.Linear` as (out, in). Why is this the most dangerous difference on the page?

pch.quizShowAnswer

B — Because copying weights without the transpose raises a shape error on a rectangular layer but SILENTLY succeeds on a square one, producing a model that runs and is wrong
A fresh 784 → 128 layer initialises at sd 0.04687 in Keras and 0.02061 in PyTorch. Is one of them wrong?

pch.quizShowAnswer

B — No — Keras defaults to glorot_uniform (scaled by fan-in and fan-out) and PyTorch to kaiming_uniform with a gain fixed for a leaky-ReLU slope of √5; they are different conventions
Both frameworks reached exactly 0.8990 test accuracy, but PyTorch took 2.1 s against Keras's 3.9 s. What is the defensible conclusion about speed?

pch.quizShowAnswer

B — On a 101,770-parameter model on CPU, per-step framework overhead dominates and PyTorch's eager loop carries less of it — a claim about this model on this hardware, not a ranking
Adam's default `epsilon` is 1e-7 in Keras and 1e-8 in PyTorch. When does that matter?

pch.quizShowAnswer

B — In float16 training, where 1e-8 underflows to zero and 1e-7 does not, so the term that was meant to guarantee a non-zero denominator stops doing so

🧪 Try It Yourself

pch.coffeeTagline

pch.coffeeCta

pch.feedbackHeading

pch.feedbackSubheading