Hyperparameter Tuning with GridSearchCV

Once cross-validation gives you a trustworthy score, the obvious next question is: which hyperparameters produce the best score? Hand-tweaking values one at a time is slow and easy to get wrong. GridSearchCVGridSearchCV automates it — you describe the grid of values to try, and it fits and cross-validates every combination for you.

What hyperparameters are

Hyperparameters are settings you choose before training.

Examples:

max_depth in a tree
C in SVM
number of neighbors in KNN

They affect bias/variance strongly.

Grid search idea

Try all combinations in a parameter grid.

diagram Diagram mermaid

  flowchart LR
  G[Parameter grid] --> CV[Cross-validation]
  CV --> S[Score combinations]
  S --> B[Best hyperparameters]

Scikit-learn example

GridSearchCV

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
 
params = {
    "C": [0.1, 1, 10],
    "gamma": ["scale", 0.01, 0.1],
    "kernel": ["rbf"],
}
 
grid = GridSearchCV(
    estimator=SVC(),
    param_grid=params,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
)
 
grid.fit(X, y)
print("best params:", grid.best_params_)
print("best score:", grid.best_score_)

GridSearchCV

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
 
params = {
    "C": [0.1, 1, 10],
    "gamma": ["scale", 0.01, 0.1],
    "kernel": ["rbf"],
}
 
grid = GridSearchCV(
    estimator=SVC(),
    param_grid=params,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
)
 
grid.fit(X, y)
print("best params:", grid.best_params_)
print("best score:", grid.best_score_)

Visualize it

Under the hood, a grid search is just a nested loop: for every cell in the grid, fit the model on each CV fold and average the score. The heatmap below sweeps that grid cell by cell — exactly the order GridSearchCVGridSearchCV evaluates them in — using the book’s own RandomForestRegressorRandomForestRegressor numbers (RMSE, lower is better) for max_featuresmax_features × n_estimatorsn_estimators. Watch the scanner land on every cell, then settle on the winner:

sketch Grid search sweeping a heatmap p5.js

A scanner sweeps every max_features x n_estimators cell in reading order, coloring it by RMSE, then glows gold on the winning combination.

Searching more than one grid at once

param_gridparam_grid doesn’t have to be a single dict — pass a list of dicts and GridSearchCVGridSearchCV evaluates each dict as its own mini grid, back to back. The book uses this to test the bootstrap=Falsebootstrap=False variant separately, since it doesn’t make sense to combine it with the default grid:

multiple_param_grids.py

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
 
param_grid = [
    {"n_estimators": [3, 10, 30], "max_features": [2, 4, 6, 8]},
    {"bootstrap": [False], "n_estimators": [3, 10], "max_features": [2, 3, 4]},
]
 
forest_reg = RandomForestRegressor(random_state=42)
grid_search = GridSearchCV(
    forest_reg,
    param_grid,
    cv=5,
    scoring="neg_mean_squared_error",
    return_train_score=True,
)
grid_search.fit(X_train, y_train)
print("best params:", grid_search.best_params_)

multiple_param_grids.py

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestRegressor
 
param_grid = [
    {"n_estimators": [3, 10, 30], "max_features": [2, 4, 6, 8]},
    {"bootstrap": [False], "n_estimators": [3, 10], "max_features": [2, 3, 4]},
]
 
forest_reg = RandomForestRegressor(random_state=42)
grid_search = GridSearchCV(
    forest_reg,
    param_grid,
    cv=5,
    scoring="neg_mean_squared_error",
    return_train_score=True,
)
grid_search.fit(X_train, y_train)
print("best params:", grid_search.best_params_)

The first dict alone is 3 × 4 = 12 combinations; the second is 1 × 2 × 3 = 6 more — 18 combinations total, each trained 5 times for cross-validation, for 90 rounds of training in one .fit().fit() call.

Reading every combination, not just the winner

grid_search.cv_results_grid_search.cv_results_ holds the score for every combination that was tried, which lets you sanity-check the whole search instead of trusting one number:

inspect_cv_results.py

import numpy as np
 
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
 
# ── Sample output (from the book) ──────────────────────────────
# 63669.05791727153 {'max_features': 2, 'n_estimators': 3}
# 55627.16171305252 {'max_features': 2, 'n_estimators': 10}
# 53384.57867637289 {'max_features': 2, 'n_estimators': 30}
# ...
# 49682.25345942335 {'max_features': 8, 'n_estimators': 30}
# ────────────────────────────────────────────────────────────────

inspect_cv_results.py

import numpy as np
 
cvres = grid_search.cv_results_
for mean_score, params in zip(cvres["mean_test_score"], cvres["params"]):
    print(np.sqrt(-mean_score), params)
 
# ── Sample output (from the book) ──────────────────────────────
# 63669.05791727153 {'max_features': 2, 'n_estimators': 3}
# 55627.16171305252 {'max_features': 2, 'n_estimators': 10}
# 53384.57867637289 {'max_features': 2, 'n_estimators': 30}
# ...
# 49682.25345942335 {'max_features': 8, 'n_estimators': 30}
# ────────────────────────────────────────────────────────────────

Because scoring="neg_mean_squared_error"scoring="neg_mean_squared_error" returns negative MSE (Scikit-learn’s convention is that higher scores are always better), you flip the sign and take the square root to get back to RMSE.

Once fitting finishes, grid_search.best_estimator_grid_search.best_estimator_ is ready to use immediately — by default GridSearchCVGridSearchCV is created with refit=Truerefit=True, so after finding the best combination via cross-validation it automatically retrains that combination on the whole training set, which usually improves performance further since it’s seeing more data. If you also passed RandomForestRegressorRandomForestRegressor as the estimator, the winning model exposes feature_importances_feature_importances_, a quick way to see which columns actually mattered to the tuned model:

feature_importances.py

feature_importances = grid_search.best_estimator_.feature_importances_
sorted(zip(feature_importances, feature_names), reverse=True)[:3]
# [(0.366, "median_income"), (0.165, "INLAND"), (0.109, "pop_per_hhold")]

feature_importances.py

feature_importances = grid_search.best_estimator_.feature_importances_
sorted(zip(feature_importances, feature_names), reverse=True)[:3]
# [(0.366, "median_income"), (0.165, "INLAND"), (0.109, "pop_per_hhold")]

Limitations

Grid search becomes expensive when:

you have many parameters
each parameter has many values

That’s where RandomizedSearchCV helps.

Mini-checkpoint

If you have 5 parameters with 10 options each:

grid size = 10^5 = 100,000 runs (too much).

🧪 Try It Yourself

Exercise 1 – Build a Parameter Grid

Exercise 2 – Fit GridSearchCV and Read the Winner

Exercise 3 – Check for an Edge-of-Grid Result

Continue to RandomizedSearchCV for Large Parameter Spaces — for when the grid gets too big to search exhaustively.

If this helped you, consider buying me a coffee ☕

Buy me a coffee