A single train/validation split gives you exactly one number, and one number can lie — get an unlucky split and a great model looks mediocre, or a badly overfit model looks fine. Cross-validation fixes this by testing your model against several different splits and reporting both the average score and how much it wobbles.

Why cross-validation

A single train/validation split can be noisy.

Cross-validation averages performance across multiple splits.

K-fold CV

Steps:

1. split data into K folds
2. train on K-1 folds
3. validate on the remaining fold
4. repeat for all folds
5. average the scores

  flowchart TD
  A[Data] --> B[Split into K folds]
  B --> C[Train on K-1]
  C --> D[Validate on 1]
  D --> E[Repeat K times]
  E --> F[Average score]

Scikit-learn example

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
 
scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5, scoring="accuracy")
print("mean:", scores.mean())
print("std:", scores.std())

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
 
scores = cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=5, scoring="accuracy")
print("mean:", scores.mean())
print("std:", scores.std())

Visualize it

K-fold CV rotates which fold is held out for validation. Over K rounds, every row of data gets used for validation exactly once, and for training K-1 times:

Stratified CV

For classification, use stratified splits so each fold preserves class ratios.

Scikit-learn does this automatically for many classifiers.

Time series warning

Don’t use random k-fold for time series.

Use time-series split.

Mini-checkpoint

When data is small:

• prefer CV over one split.

🧪 Try It Yourself

Exercise 1 – Run 5-Fold Cross-Validation

Exercise 2 – Report Mean and Standard Deviation

Exercise 3 – Convert Negative MSE Scoring to RMSE

Next

Continue to Hyperparameter Tuning with GridSearchCV — now that you can measure a model reliably, use that measurement to search for the best hyperparameters.