What you’ll learn

• how much a reported score moves purely by changing the random seed — measured
• the k-fold procedure, and why every row is validated exactly once
• how to choose $k$, with the bias-variance argument behind the usual answer of 5 or 10
• the standard deviation across folds, which is the number people forget to report
• four splitters — plain, stratified, grouped and time-series — and when each is mandatory
• cross_validatecross_validate for several metrics and train scores in one pass

Intuition

You split 80/20, train, score, and report 0.9737.

Change random_staterandom_state from 0 to 1 and rerun. Now it is 0.9561. Neither number is wrong; both are measurements of the same model on different samples of the same data. Reporting either alone presents a coin flip as a fact.

Cross-validation replaces one measurement with $k$ of them, each on a different fifth of the data, and reports the mean. It also gives you something a single split never can: a spread, which is how you know whether a difference between two models is real.

figure5-fold cross-validationmatplotlib

How much does a single split move?

figureThe same model and data, 60 different random seedsmatplotlib

Reading the plot

1. The single split reaches 1.0000 on some seeds. Perfect accuracy, from a model that averages 0.978. Pick that seed and report it and you are not lying, exactly — but nobody could reproduce it.
2. The spread is 5.3 points wide. Two models differing by 2 points cannot be distinguished on a single split of this dataset.
3. CV reduces the standard deviation by 3.5×. Not by being cleverer, but by averaging five measurements instead of taking one — the same $1/\sqrt{k}$ that governs any mean.
4. The means agree (0.9785 against 0.9773). CV is not more optimistic or more pessimistic; it is more precise.

The math

Given $k$ fold scores $s_1, \ldots, s_k$:

Report both. “0.85 ± 0.07” and “0.85 ± 0.005” describe very different situations, and only the first tells you a rival at 0.87 might be noise.

Choosing k

$k$ trades three things against each other:

Small $k$ means each model trains on less data, so it underperforms the model you will actually ship — a pessimistic bias. Large $k$ removes that bias, but the $k$ training sets then overlap almost completely, so their scores are highly correlated and averaging them removes less noise than you would expect.

Leave-one-out is the surprise. It sounds like the gold standard and is usually a poor choice: $n$ fits, and $n$ nearly identical models, so the variance of the estimate stays high. Use it only when $n$ is tiny.

5 or 10 is the answer for almost everything, and 5 is fine.

Worked example by hand

Five fold scores from a classifier:

Step 1 — the mean.

Step 2 — the deviations and their squares.

Step 3 — the standard deviation.

Step 4 — report it. $0.850 \pm 0.071$.

Step 5 — read it. The folds range from 0.75 to 0.95, a 20-point swing on the same model. A rival model scoring 0.87 is not better on this evidence. If you need to distinguish models at that resolution, you need more data or more folds — not a bolder claim.

In code

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
 
X, y = load_breast_cancer(return_X_y=True)
 
# The pipeline matters: the scaler is refitted inside each fold
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
 
scores = cross_val_score(model, X, y, cv=5)
print(scores.round(4))                    # one score per fold
print(f"{scores.mean():.4f} +/- {scores.std():.4f}")

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
 
X, y = load_breast_cancer(return_X_y=True)
 
# The pipeline matters: the scaler is refitted inside each fold
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
 
scores = cross_val_score(model, X, y, cv=5)
print(scores.round(4))                    # one score per fold
print(f"{scores.mean():.4f} +/- {scores.std():.4f}")

Several metrics, and train scores, in one pass

from sklearn.model_selection import cross_validate
 
results = cross_validate(
    model, X, y, cv=5,
    scoring=["accuracy", "precision", "recall", "roc_auc"],
    return_train_score=True,
)
 
for name in ("accuracy", "precision", "recall", "roc_auc"):
    train = results[f"train_{name}"].mean()
    test = results[f"test_{name}"].mean()
    print(f"{name:<10} train {train:.4f}   test {test:.4f}   gap {train - test:+.4f}")
 
# accuracy   train 0.9895   test 0.9807   gap +0.0088
# precision  train 0.9862   test 0.9782   gap +0.0080
# recall     train 0.9972   test 0.9916   gap +0.0056
# roc_auc    train 0.9977   test 0.9952   gap +0.0025

from sklearn.model_selection import cross_validate
 
results = cross_validate(
    model, X, y, cv=5,
    scoring=["accuracy", "precision", "recall", "roc_auc"],
    return_train_score=True,
)
 
for name in ("accuracy", "precision", "recall", "roc_auc"):
    train = results[f"train_{name}"].mean()
    test = results[f"test_{name}"].mean()
    print(f"{name:<10} train {train:.4f}   test {test:.4f}   gap {train - test:+.4f}")
 
# accuracy   train 0.9895   test 0.9807   gap +0.0088
# precision  train 0.9862   test 0.9782   gap +0.0080
# recall     train 0.9972   test 0.9916   gap +0.0056
# roc_auc    train 0.9977   test 0.9952   gap +0.0025

return_train_score=Truereturn_train_score=True gives you the overfitting diagnosis from Underfitting vs Overfitting for free, on every metric, in the same call. Gaps under 0.01 here — a well-behaved model.

See it move

The validation block rotates through the dataset. Each pass produces one score; the running mean and spread build up on the right.

Four splitters

figureBlue trains, amber validatesmatplotlib

Choosing between them is mechanical once you know what your rows are:

  flowchart TD
  A["What is one row?"] --> B{"Do rows have
a time order that
matters?"}
  B -->|"yes -- events, readings,
transactions"| C["TimeSeriesSplit"]
  C --> C2["Folds are not comparable:
fold 0 trains on 4 rows,
fold 3 on 16."]
  B -->|"no"| D{"Can several rows
belong to the same
entity?"}
  D -->|"yes -- patients, users,
devices, documents"| E["GroupKFold
(or StratifiedGroupKFold)"]
  E --> E2["A group on both sides
lets the model recognise
the entity, not the pattern."]
  D -->|"no, rows are independent"| F{"Classification?"}
  F -->|"yes"| G["StratifiedKFold
shuffle=True"]
  F -->|"no, regression"| H{"Is the row order
meaningful in the file?"}
  H -->|"could be -- sorted by
date, region, id"| I["KFold shuffle=True,
fixed random_state"]
  H -->|"definitely not"| J["KFold, the default"]
  G --> K["Repeat with several seeds
if the fold spread is wide."]
  I --> K
  J --> K

Two of those branches are guards against leakage rather than preferences, which is why the table calls them mandatory: a group spanning folds and a future row in the training set both produce a score that is higher than the truth and gives no warning.

Time series

A random split lets the model train on Thursday to predict Tuesday. The score is excellent and entirely meaningless.

from sklearn.model_selection import TimeSeriesSplit
 
splitter = TimeSeriesSplit(n_splits=4)
for i, (train_idx, val_idx) in enumerate(splitter.split(range(20))):
    print(f"fold {i}: train {train_idx.min()}-{train_idx.max()}"
          f"   validate {val_idx.min()}-{val_idx.max()}")
 
# fold 0: train 0-3    validate 4-7
# fold 1: train 0-7    validate 8-11
# fold 2: train 0-11   validate 12-15
# fold 3: train 0-15   validate 16-19

from sklearn.model_selection import TimeSeriesSplit
 
splitter = TimeSeriesSplit(n_splits=4)
for i, (train_idx, val_idx) in enumerate(splitter.split(range(20))):
    print(f"fold {i}: train {train_idx.min()}-{train_idx.max()}"
          f"   validate {val_idx.min()}-{val_idx.max()}")
 
# fold 0: train 0-3    validate 4-7
# fold 1: train 0-7    validate 8-11
# fold 2: train 0-11   validate 12-15
# fold 3: train 0-15   validate 16-19

The training window grows and validation always sits immediately after it. Note the consequence: fold 0 trains on 4 rows and fold 3 on 16, so the folds are not comparable to each other. That is the price of respecting time.

Groups

One row per hospital visit, several visits per patient. A random split puts the same patient on both sides, the model recognises the patient rather than the condition, and the score is inflated.

import numpy as np
from sklearn.model_selection import GroupKFold, cross_val_score
 
groups = np.repeat(np.arange(100), 5)      # 100 patients, 5 visits each
 
naive = cross_val_score(model, X_visits, y_visits, cv=5).mean()
honest = cross_val_score(model, X_visits, y_visits,
                         cv=GroupKFold(5), groups=groups).mean()
 
print(f"random split: {naive:.4f}")        # optimistic
print(f"group split:  {honest:.4f}")       # the number that transfers

import numpy as np
from sklearn.model_selection import GroupKFold, cross_val_score
 
groups = np.repeat(np.arange(100), 5)      # 100 patients, 5 visits each
 
naive = cross_val_score(model, X_visits, y_visits, cv=5).mean()
honest = cross_val_score(model, X_visits, y_visits,
                         cv=GroupKFold(5), groups=groups).mean()
 
print(f"random split: {naive:.4f}")        # optimistic
print(f"group split:  {honest:.4f}")       # the number that transfers

Pitfalls

🧪 Try It Yourself

Exercise 1 – Mean and spread by hand

Exercise 2 – Run 5-fold cross-validation

Exercise 3 – Measure the noise a single split adds

Exercise 4 – Several metrics in one pass

Exercise 5 – Respect the arrow of time

Recap

• A single 80/20 split on this dataset reports anywhere from 0.9474 to 1.0000 across seeds. The 5-fold mean spans 0.9684 to 0.9842, with 3.5 times less spread.
• Every row is validated exactly once; you get $k$ scores, a mean and a standard deviation.
• Report both. $0.850 \pm 0.071$ and $0.850 \pm 0.005$ demand different conclusions.
• $k = 5$ or 10. Leave-one-out is expensive and, because its training sets overlap almost entirely, higher-variance than it sounds.
• Four splitters: plain, stratified (any classification), grouped (repeated measures), time-series (anything temporal). cv=5cv=5 does not shuffle.
• cross_validatecross_validate returns several metrics plus train scores in one pass.
• Always pass the pipeline, never a pre-transformed matrix.

Exercise 6 – What does raising the fold count buy?

Next

Continue to Hyperparameter Tuning with GridSearchCV — use this machinery not just to measure a model but to choose between hundreds of them.