K-Fold Cross-Validation

What you’ll learn

• how much a reported score moves purely by changing the random seed — measured
• the k-fold procedure, and why every row is validated exactly once
• how to choose $k$, with the bias-variance argument behind the usual answer of 5 or 10
• the standard deviation across folds, which is the number people forget to report
• four splitters — plain, stratified, grouped and time-series — and when each is mandatory
• cross_validate for several metrics and train scores in one pass

Intuition

You split 80/20, train, score, and report 0.9737.

Change random_state from 0 to 1 and rerun. Now it is 0.9561. Neither number is wrong; both are measurements of the same model on different samples of the same data. Reporting either alone presents a coin flip as a fact.

Cross-validation replaces one measurement with $k$ of them, each on a different fifth of the data, and reports the mean. It also gives you something a single split never can: a spread, which is how you know whether a difference between two models is real.

figure 5-fold cross-validation matplotlib

Every row is validated exactly once and trained on four times. Five scores come out, and both their mean and their spread are meaningful.

How much does a single split move?

figure The same model and data, 60 different random seeds matplotlib

A single 80/20 split reports anywhere from 0.9474 to 1.0000 — a spread of 5.3 points. The 5-fold mean spans 0.9684 to 0.9842, and its standard deviation is 3.5 times smaller.

	Mean	Std dev	Observed range
Single 80/20 split	0.9785	0.0118	0.9474 – 1.0000
5-fold CV mean	0.9773	0.0034	0.9684 – 0.9842

Reading the plot

1. The single split reaches 1.0000 on some seeds. Perfect accuracy, from a model that averages 0.978. Pick that seed and report it and you are not lying, exactly — but nobody could reproduce it.
2. The spread is 5.3 points wide. Two models differing by 2 points cannot be distinguished on a single split of this dataset.
3. CV reduces the standard deviation by 3.5×. Not by being cleverer, but by averaging five measurements instead of taking one — the same $1/\sqrt{k}$ that governs any mean.
4. The means agree (0.9785 against 0.9773). CV is not more optimistic or more pessimistic; it is more precise.

The math

Given $k$ fold scores $s_1, \ldots, s_k$:

$$\bar{s} = \frac{1}{k}\sum_{i=1}^{k} s_i \qquad \text{sd} = \sqrt{\frac{1}{k}\sum_{i=1}^{k}\left(s_i - \bar{s}\right)^2}$$

Report both. “0.85 ± 0.07” and “0.85 ± 0.005” describe very different situations, and only the first tells you a rival at 0.87 might be noise.

Choosing k

$k$ trades three things against each other:

$k$	Training set per fold	Bias of the estimate	Variance of the estimate	Cost
2	50%	High — trained on half the data	Low	2 fits
5	80%	Small	Moderate	5 fits
10	90%	Smaller	Moderate	10 fits
$n$ (leave-one-out)	$n-1$	Almost none	High	$n$ fits

Small $k$ means each model trains on less data, so it underperforms the model you will actually ship — a pessimistic bias. Large $k$ removes that bias, but the $k$ training sets then overlap almost completely, so their scores are highly correlated and averaging them removes less noise than you would expect.

Leave-one-out is the surprise. It sounds like the gold standard and is usually a poor choice: $n$ fits, and $n$ nearly identical models, so the variance of the estimate stays high. Use it only when $n$ is tiny.

5 or 10 is the answer for almost everything, and 5 is fine.

Worked example by hand

Five fold scores from a classifier:

$$[\,0.80,\; 0.90,\; 0.85,\; 0.95,\; 0.75\,]$$

Step 1 — the mean.

$$\bar{s} = \frac{0.80 + 0.90 + 0.85 + 0.95 + 0.75}{5} = \frac{4.25}{5} = 0.85$$

Step 2 — the deviations and their squares.

fold	$s_i$	$s_i - \bar{s}$	$(s_i - \bar{s})^2$
1	0.80	−0.05	0.0025
2	0.90	+0.05	0.0025
3	0.85	0.00	0.0000
4	0.95	+0.10	0.0100
5	0.75	−0.10	0.0100
		0.00	0.0250

Step 3 — the standard deviation.

$$\text{sd} = \sqrt{\frac{0.0250}{5}} = \sqrt{0.005} \approx 0.0707$$

Step 4 — report it. $0.850 \pm 0.071$.

Step 5 — read it. The folds range from 0.75 to 0.95, a 20-point swing on the same model. A rival model scoring 0.87 is not better on this evidence. If you need to distinguish models at that resolution, you need more data or more folds — not a bolder claim.

In code

basic_cv.py

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
 
X, y = load_breast_cancer(return_X_y=True)
 
# The pipeline matters: the scaler is refitted inside each fold
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
 
scores = cross_val_score(model, X, y, cv=5)
print(scores.round(4))                    # one score per fold
print(f"{scores.mean():.4f} +/- {scores.std():.4f}")

Cross-validating without a pipeline is a leak

cross_val_score(LogisticRegression(), StandardScaler().fit_transform(X), y) scales using statistics from every row, including each fold’s validation rows. The score comes out optimistic. Pass the pipeline, not the pre-transformed matrix — Transformation Pipelines shows the damage this does.

Several metrics, and train scores, in one pass

cross_validate.py

from sklearn.model_selection import cross_validate
 
results = cross_validate(
    model, X, y, cv=5,
    scoring=["accuracy", "precision", "recall", "roc_auc"],
    return_train_score=True,
)
 
for name in ("accuracy", "precision", "recall", "roc_auc"):
    train = results[f"train_{name}"].mean()
    test = results[f"test_{name}"].mean()
    print(f"{name:<10} train {train:.4f}   test {test:.4f}   gap {train - test:+.4f}")
 
# accuracy   train 0.9895   test 0.9807   gap +0.0088
# precision  train 0.9862   test 0.9782   gap +0.0080
# recall     train 0.9972   test 0.9916   gap +0.0056
# roc_auc    train 0.9977   test 0.9952   gap +0.0025

return_train_score=True gives you the overfitting diagnosis from Underfitting vs Overfitting for free, on every metric, in the same call. Gaps under 0.01 here — a well-behaved model.

See it move

The validation block rotates through the dataset. Each pass produces one score; the running mean and spread build up on the right.

sketch Five folds, five scores, one estimate p5.js Pause Restart

The amber validation block rotates through the rows. Each fold contributes a score, and the running mean plus spread settle as the folds accumulate.

Four splitters

figure Blue trains, amber validates matplotlib

KFold takes contiguous blocks. StratifiedKFold preserves the class ratio. GroupKFold keeps whole groups together. TimeSeriesSplit never lets training data come from after validation data.

Splitter	Guarantees	Mandatory when
KFold	Every row validated once	The default, for i.i.d. data
StratifiedKFold	Class ratio preserved per fold	Any classification, especially imbalanced
GroupKFold	A group never spans folds	Repeated measures — patients, users, devices
TimeSeriesSplit	Training always precedes validation	Any temporal ordering

Choosing between them is mechanical once you know what your rows are:

diagram Diagram mermaid

  flowchart TD
  A["What is one row?"] --> B{"Do rows have
a time order that
matters?"}
  B -->|"yes -- events, readings,
transactions"| C["TimeSeriesSplit"]
  C --> C2["Folds are not comparable:
fold 0 trains on 4 rows,
fold 3 on 16."]
  B -->|"no"| D{"Can several rows
belong to the same
entity?"}
  D -->|"yes -- patients, users,
devices, documents"| E["GroupKFold
(or StratifiedGroupKFold)"]
  E --> E2["A group on both sides
lets the model recognise
the entity, not the pattern."]
  D -->|"no, rows are independent"| F{"Classification?"}
  F -->|"yes"| G["StratifiedKFold
shuffle=True"]
  F -->|"no, regression"| H{"Is the row order
meaningful in the file?"}
  H -->|"could be -- sorted by
date, region, id"| I["KFold shuffle=True,
fixed random_state"]
  H -->|"definitely not"| J["KFold, the default"]
  G --> K["Repeat with several seeds
if the fold spread is wide."]
  I --> K
  J --> K

Two of those branches are guards against leakage rather than preferences, which is why the table calls them mandatory: a group spanning folds and a future row in the training set both produce a score that is higher than the truth and gives no warning.

cross_val_score picks a splitter for you

Passing cv=5 gives StratifiedKFold for classifiers and plain KFold for regressors — and neither shuffles. On data sorted by class or by date, unshuffled folds are catastrophic. Pass an explicit splitter with shuffle=True whenever the row order might carry meaning.

Time series

A random split lets the model train on Thursday to predict Tuesday. The score is excellent and entirely meaningless.

time_series_cv.py

from sklearn.model_selection import TimeSeriesSplit
 
splitter = TimeSeriesSplit(n_splits=4)
for i, (train_idx, val_idx) in enumerate(splitter.split(range(20))):
    print(f"fold {i}: train {train_idx.min()}-{train_idx.max()}"
          f"   validate {val_idx.min()}-{val_idx.max()}")
 
# fold 0: train 0-3    validate 4-7
# fold 1: train 0-7    validate 8-11
# fold 2: train 0-11   validate 12-15
# fold 3: train 0-15   validate 16-19

The training window grows and validation always sits immediately after it. Note the consequence: fold 0 trains on 4 rows and fold 3 on 16, so the folds are not comparable to each other. That is the price of respecting time.

Groups

One row per hospital visit, several visits per patient. A random split puts the same patient on both sides, the model recognises the patient rather than the condition, and the score is inflated.

group_cv.py

import numpy as np
from sklearn.model_selection import GroupKFold, cross_val_score
 
groups = np.repeat(np.arange(100), 5)      # 100 patients, 5 visits each
 
naive = cross_val_score(model, X_visits, y_visits, cv=5).mean()
honest = cross_val_score(model, X_visits, y_visits,
                         cv=GroupKFold(5), groups=groups).mean()
 
print(f"random split: {naive:.4f}")        # optimistic
print(f"group split:  {honest:.4f}")       # the number that transfers

Pitfalls

Preprocessing outside the loop

Every transformation that learns from data must sit inside the pipeline you hand to cross_val_score. Otherwise each fold’s validation rows helped choose the parameters used to transform them.

Reporting the mean without the spread

“0.85” hides whether the folds were 0.84–0.86 or 0.75–0.95. The second means most apparent improvements you chase will be noise.

Unshuffled folds on ordered data

KFold takes contiguous blocks. If the CSV is sorted by class, fold 1 may contain a single class and score zero. shuffle=True is one keyword and prevents an entire category of confusing results.

Using CV scores as your final number

CV is for choosing between options. Once you have tuned against it, it is optimistic for the same reason a reused test set is — see the nested cross-validation section in Hyperparameter Tuning with GridSearchCV.

pch.quizTag pch.quizDefaultTitle

1. On this dataset a single 80/20 split reports between 0.9474 and 1.0000 depending on the seed. What does that imply?

   A The model is unstable and should be replaced B A single split cannot distinguish two models differing by less than about 5 points, so its score should never be reported alone C The dataset is corrupted D You should choose the seed that gives the best score
   pch.quizShowAnswer

   B — A single split cannot distinguish two models differing by less than about 5 points, so its score should never be reported alone — The model is fine; the measurement is noisy. Averaging five folds cuts that noise by 3.5 times and yields a number that reproduces.

2. Why is leave-one-out cross-validation usually a poor choice despite its low bias?

   A It is biased toward optimistic scores B The n training sets overlap almost completely, so the n scores are highly correlated and averaging them removes little noise — at n times the cost C It cannot be used for classification D It requires shuffling
   pch.quizShowAnswer

   B — The n training sets overlap almost completely, so the n scores are highly correlated and averaging them removes little noise — at n times the cost — Low bias, high variance, high cost. Five or ten folds give a better estimate for a fraction of the compute.

3. You have several rows per patient. Which splitter do you need?

   A KFold with shuffle B StratifiedKFold C GroupKFold with patient as the group D TimeSeriesSplit
   pch.quizShowAnswer

   C — GroupKFold with patient as the group — Without grouping, a patient's other visits sit in the training set and the model can identify the patient rather than learn the condition. GroupKFold keeps every patient wholly on one side.

4. Your five folds score 0.80, 0.90, 0.85, 0.95 and 0.75. A rival model reports 0.87. What can you conclude?

   A The rival is better B Nothing — your own folds span 20 points with a standard deviation of 0.071, so 0.87 is well inside the noise C Your model is better D You should rerun with a different seed
   pch.quizShowAnswer

   B — Nothing — your own folds span 20 points with a standard deviation of 0.071, so 0.87 is well inside the noise — 0.850 ± 0.071 comfortably contains 0.87. Distinguishing models at that resolution needs more data or more folds, not a bolder claim.

🧪 Try It Yourself

Exercise 1 – Mean and spread by hand

pch.exerciseTag pch.exerciseTitle

Exercise 2 – Run 5-fold cross-validation

pch.exerciseTag pch.exerciseTitle

Exercise 3 – Measure the noise a single split adds

pch.exerciseTag pch.exerciseTitle

Exercise 4 – Several metrics in one pass

pch.exerciseTag pch.exerciseTitle

Exercise 5 – Respect the arrow of time

pch.exerciseTag pch.exerciseTitle

Recap

• A single 80/20 split on this dataset reports anywhere from 0.9474 to 1.0000 across seeds. The 5-fold mean spans 0.9684 to 0.9842, with 3.5 times less spread.
• Every row is validated exactly once; you get $k$ scores, a mean and a standard deviation.
• Report both. $0.850 \pm 0.071$ and $0.850 \pm 0.005$ demand different conclusions.
• $k = 5$ or 10. Leave-one-out is expensive and, because its training sets overlap almost entirely, higher-variance than it sounds.
• Four splitters: plain, stratified (any classification), grouped (repeated measures), time-series (anything temporal). cv=5 does not shuffle.
• cross_validate returns several metrics plus train scores in one pass.
• Always pass the pipeline, never a pre-transformed matrix.

Exercise 6 – What does raising the fold count buy?

pch.exerciseTag pch.exerciseTitle

Next

Continue to Hyperparameter Tuning with GridSearchCV — use this machinery not just to measure a model but to choose between hundreds of them.