What you’ll learn

• why you need three data splits (train, validation, test), not two
• information leaks — how tuning on the validation set slowly corrupts it
• simple hold-out validation vs K-fold vs iterated K-fold
• the canonical overfitting curve: training loss down, validation loss turns up
• three common evaluation mistakes: unshuffled data, time leaks, and duplicate rows

Why not just train/test?

The obvious approach — split data into training and test sets, train on one, score on the other — has a hidden flaw. Developing a model always means tuning its configuration: how many layers, how wide, which learning rate. You choose those settings by watching performance on some held-out data. That watching is a form of learning — a search through configuration space — so if that held-out data is your test set, you’ll unconsciously overfit to it, one small decision at a time.

The fix is a third split: keep a validation set for every tuning decision during development, and a test set you touch exactly once, at the very end.

  flowchart LR
  D["All labeled data"] --> T["Training set
(fit the weights)"]
  D --> V["Validation set
(tune everything:
architecture, hyperparameters)"]
  D --> S["Test set
(touch once, at the end)"]

Information leaks

Every time you tune a hyperparameter based on validation performance, a tiny bit of information about the validation set leaks into the model. Do this once and it’s harmless. Do it hundreds of times across a long tuning campaign, and your model quietly overfits to the validation process itself — even though it was never directly trained on that data. That’s exactly why the test set has to stay untouched until the very end: it’s your only unbiased read on how the model will behave on truly new data.

Simple hold-out validation

Set aside a chunk of data as validation, train on the rest:

import numpy as np
 
data = np.arange(20)
np.random.seed(0)
np.random.shuffle(data)
 
num_validation_samples = 4
validation_data = data[:num_validation_samples]
training_data = data[num_validation_samples:]
 
print("train size:", len(training_data), "val size:", len(validation_data))

import numpy as np
 
data = np.arange(20)
np.random.seed(0)
np.random.shuffle(data)
 
num_validation_samples = 4
validation_data = data[:num_validation_samples]
training_data = data[num_validation_samples:]
 
print("train size:", len(training_data), "val size:", len(validation_data))

Simple and fast — but if you don’t have much data, your validation set may be too small to be statistically representative. A telltale sign: reshuffling the data before splitting gives you noticeably different validation scores each time.

K-fold validation

Split the data into KK equal partitions. For each partition, train on the other K - 1K - 1 and validate on the one left out; the final score is the average across all KK runs. This is the right call once a single hold-out split feels unstable.

  flowchart TD
  D["Full dataset
(shuffled)"] --> F1["Fold 1: validation
Folds 2, 3: train"]
  D --> F2["Fold 2: validation
Folds 1, 3: train"]
  D --> F3["Fold 3: validation
Folds 1, 2: train"]
  F1 --> S["Average the
3 validation scores"]
  F2 --> S
  F3 --> S

import numpy as np
from sklearn.model_selection import KFold
 
X = np.arange(12).reshape(6, 2)
kf = KFold(n_splits=3)
 
for fold, (train_idx, val_idx) in enumerate(kf.split(X)):
    print(f"fold {fold}: train={len(train_idx)} val={len(val_idx)}")

import numpy as np
from sklearn.model_selection import KFold
 
X = np.arange(12).reshape(6, 2)
kf = KFold(n_splits=3)
 
for fold, (train_idx, val_idx) in enumerate(kf.split(X)):
    print(f"fold {fold}: train={len(train_idx)} val={len(val_idx)}")

Iterated K-fold validation

For the rare case where you have very little data and need the most reliable score possible: run K-fold validation PP times, reshuffling the data before each run, and average all P × KP × K scores. It’s expensive — you train that many models — but it’s the gold standard when a single K-fold run still looks noisy.

import numpy as np
 
# each entry: the mean validation score from one full K-fold run
run_scores = [0.81, 0.83, 0.80]
iterated_kfold_score = np.mean(run_scores)
print(round(iterated_kfold_score, 3))

import numpy as np
 
# each entry: the mean validation score from one full K-fold run
run_scores = [0.81, 0.83, 0.80]
iterated_kfold_score = np.mean(run_scores)
print(round(iterated_kfold_score, 3))

Three ways to leak information without noticing

• Unshuffled data. If your samples are ordered by class and you slice off the last 20% as your test set, you may end up training only on classes 0–7 and testing only on classes 8–9. Always shuffle before splitting — unless…
• The arrow of time. For time-ordered data (tomorrow’s weather, stock prices), shuffling is exactly wrong: it lets the model train on the future and get “tested” on the past. Test data must always come after training data chronologically.
• Redundant rows. If the same sample (or a near-duplicate) appears in both the training and validation sets, you’re partly validating on data the model has already memorized. Deduplicate before splitting.

Visualize it: the canonical overfitting curve

Every model type, on every dataset, tends to produce the same shape: training loss falls smoothly and keeps falling, while validation loss falls too — until it hits a floor and starts climbing back up. That turning point is the best generalizing version of the model you’re going to get; training past it only makes things worse.

🧪 Try It Yourself

Exercise 1 – Simple Holdout Split

Exercise 2 – Set Up 3-Fold Cross-Validation

Exercise 3 – Spot a Data Representativeness Bug

Next

Continue to Callbacks and TensorBoard — now that you can trust your validation score, use it to automatically stop training, save the best model, and watch both curves live instead of guessing after the fact.