RandomizedSearchCV for Large Parameter Spaces

Grid search is exhaustive, which is exactly the problem once you have more than a couple of hyperparameters: the number of combinations multiplies fast, and most of your compute budget gets spent evaluating near-duplicate settings. Randomized search trades completeness for coverage — it samples a fixed number of combinations and almost always finds something close to the best, for a fraction of the cost.

Why randomized search

Randomized search samples a fixed number of combinations from distributions.

It’s often more efficient than grid search because:

not all hyperparameters matter equally
you can explore wide ranges quickly

The idea

diagram Diagram mermaid

  flowchart LR
  D[Parameter distributions] --> R[Sample N combos]
  R --> CV[Cross-validation]
  CV --> B[Best found]

Scikit-learn example

RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
 
params = {
    "n_estimators": [200, 400, 800],
    "max_depth": [None, 5, 10, 20],
    "min_samples_leaf": [1, 2, 4, 8],
}
 
search = RandomizedSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_distributions=params,
    n_iter=20,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
    random_state=42,
)
 
search.fit(X, y)
print("best params:", search.best_params_)
print("best score:", search.best_score_)

RandomizedSearchCV

from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
 
params = {
    "n_estimators": [200, 400, 800],
    "max_depth": [None, 5, 10, 20],
    "min_samples_leaf": [1, 2, 4, 8],
}
 
search = RandomizedSearchCV(
    estimator=RandomForestClassifier(random_state=42),
    param_distributions=params,
    n_iter=20,
    cv=5,
    scoring="accuracy",
    n_jobs=-1,
    random_state=42,
)
 
search.fit(X, y)
print("best params:", search.best_params_)
print("best score:", search.best_score_)

Visualize it

Where GridSearchCVGridSearchCV sweeps every cell of a fixed grid in order, RandomizedSearchCVRandomizedSearchCV throws darts at random points across the (continuous) hyperparameter space. Watch each random draw land, and notice the gold “best so far” marker jump only when a new draw beats every previous one — after n_itern_iter draws it settles wherever the best sample happened to land, not on a rigid grid line:

sketch Random search scattering the parameter space p5.js

Each random draw appears one at a time inside a continuous 2D hyperparameter space; a gold ring tracks the best score found so far.

Sampling from distributions, not just lists

param_distributionsparam_distributions doesn’t have to be a list of fixed values — you can hand it a scipy.statsscipy.stats distribution and RandomizedSearchCVRandomizedSearchCV will draw a fresh random value from it on every iteration. This is where randomized search really pulls ahead of grid search: a grid can only ever test the handful of values you typed in, but a distribution can be sampled at any point in its range.

param_distributions_with_scipy.py

from scipy.stats import randint, uniform
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestRegressor
 
param_distribs = {
    "n_estimators": randint(low=1, high=200),      # any integer from 1-199
    "max_features": randint(low=1, high=8),        # any integer from 1-7
    "min_samples_leaf": randint(low=1, high=10),
    "bootstrap": [True, False],                     # plain lists still work too
}
 
forest_reg = RandomForestRegressor(random_state=42)
rnd_search = RandomizedSearchCV(
    forest_reg,
    param_distributions=param_distribs,
    n_iter=10,
    cv=5,
    scoring="neg_mean_squared_error",
    random_state=42,
)
rnd_search.fit(X_train, y_train)
print("best params:", rnd_search.best_params_)

param_distributions_with_scipy.py

from scipy.stats import randint, uniform
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestRegressor
 
param_distribs = {
    "n_estimators": randint(low=1, high=200),      # any integer from 1-199
    "max_features": randint(low=1, high=8),        # any integer from 1-7
    "min_samples_leaf": randint(low=1, high=10),
    "bootstrap": [True, False],                     # plain lists still work too
}
 
forest_reg = RandomForestRegressor(random_state=42)
rnd_search = RandomizedSearchCV(
    forest_reg,
    param_distributions=param_distribs,
    n_iter=10,
    cv=5,
    scoring="neg_mean_squared_error",
    random_state=42,
)
rnd_search.fit(X_train, y_train)
print("best params:", rnd_search.best_params_)

As the book puts it, letting the search run for 1,000 iterations explores 1,000 different values per hyperparameter — versus only the handful you could realistically type into a param_gridparam_grid list. And because the budget is just the n_itern_iter argument, you control compute cost directly instead of having to prune a grid down by hand.

When to use which

GridSearchCV: small search space
RandomizedSearchCV: large/continuous search space

Mini-checkpoint

If you’re tuning learning rate in [1e-4, 1], randomized search is usually better.

🧪 Try It Yourself

Exercise 1 – Compare Grid Size vs Search Budget

Exercise 2 – Fit RandomizedSearchCV

Exercise 3 – Choose the Right Search Strategy

Continue to The ML Pipeline: Automating the Workflow — wrap preprocessing and the tuned model into a single object so this all stays reproducible in production.

If this helped you, consider buying me a coffee ☕

Buy me a coffee