Skip to content
Python Central Hub

Gradient Boosting (XGBoost, LightGBM, CatBoost)

What you’ll learn

  • how Gradient Boosting differs from AdaBoost: it fits the residual errors, not reweighted instances
  • how to hand-build a 3-tree Gradient Boosted ensemble and confirm it matches scikit-learn’s GradientBoostingRegressorGradientBoostingRegressor exactly
  • learning_ratelearning_rate / shrinkage, and why fewer trees at a high rate can overfit just as easily as too many trees at a low rate
  • early stopping with staged_predict()staged_predict()
  • where HistGradientBoostingClassifierHistGradientBoostingClassifier, XGBoost, LightGBM, and CatBoost fit in

What gradient boosting is

Just like AdaBoost, Gradient Boosting builds an ensemble sequentially, each new predictor trying to improve on the one before it. The difference is how it corrects its predecessor: instead of reweighting misclassified instances, Gradient Boosting trains the next tree to fit the residual errors — literally, what’s left over after subtracting the current ensemble’s predictions from the true targets.

text
prediction = tree1 + tree2 + tree3 + ...
text
prediction = tree1 + tree2 + tree3 + ...

Géron calls the regression version of this Gradient Tree Boosting, or Gradient Boosted Regression Trees (GBRT).

diagram Gradient boosting: fit the residuals mermaid
Each new tree is trained to predict what the current ensemble still gets wrong.

Building it by hand

The clearest way to understand GBRT is to build one manually, exactly as Géron does in Chapter 7, on a small noisy quadratic dataset.

Three hand-trained trees, chained together
import numpy as np
from sklearn.tree import DecisionTreeRegressor
 
np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(100)
 
# Tree 1: fit the original targets
tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg1.fit(X, y)
 
# Tree 2: fit tree 1's residual errors
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg2.fit(X, y2)
 
# Tree 3: fit tree 2's residual errors
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg3.fit(X, y3)
 
X_new = np.array([[0.0], [0.2], [-0.4]])
y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))
print([round(float(v), 3) for v in y_pred])
Three hand-trained trees, chained together
import numpy as np
from sklearn.tree import DecisionTreeRegressor
 
np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(100)
 
# Tree 1: fit the original targets
tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg1.fit(X, y)
 
# Tree 2: fit tree 1's residual errors
y2 = y - tree_reg1.predict(X)
tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg2.fit(X, y2)
 
# Tree 3: fit tree 2's residual errors
y3 = y2 - tree_reg2.predict(X)
tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)
tree_reg3.fit(X, y3)
 
X_new = np.array([[0.0], [0.2], [-0.4]])
y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))
print([round(float(v), 3) for v in y_pred])
text
[0.04, 0.171, 0.495]
text
[0.04, 0.171, 0.495]

Now compare against scikit-learn’s built-in GradientBoostingRegressorGradientBoostingRegressor, using the exact same data, tree depth, and a learning_rate=1.0learning_rate=1.0 (no shrinkage):

Same ensemble, scikit-learn's way
from sklearn.ensemble import GradientBoostingRegressor
 
gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0, random_state=42)
gbrt.fit(X, y)
print([round(float(v), 3) for v in gbrt.predict(X_new)])
Same ensemble, scikit-learn's way
from sklearn.ensemble import GradientBoostingRegressor
 
gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3, learning_rate=1.0, random_state=42)
gbrt.fit(X, y)
print([round(float(v), 3) for v in gbrt.predict(X_new)])
text
[0.04, 0.171, 0.495]
text
[0.04, 0.171, 0.495]

Identical — GradientBoostingRegressorGradientBoostingRegressor is doing exactly what we just did by hand, tree by tree.

Shrinkage: the learning_ratelearning_rate hyperparameter

learning_ratelearning_rate scales down each tree’s contribution before it’s added to the ensemble. A low value (e.g. 0.10.1) needs more trees to fit the training data, but usually generalizes better — this regularization trick is called shrinkage. Get the balance wrong and you’ll see the classic under/overfitting pair: too few trees at a low learning rate underfits, too many overfits.

Early stopping with staged_predict()staged_predict()

Rather than guessing n_estimatorsn_estimators, train a generously large ensemble once, then walk back through its staged_predict()staged_predict() iterator (predictions after 1 tree, after 2 trees, after 3 trees, …) to find exactly where validation error is lowest.

Finding the best number of trees
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
np.random.seed(42)
X = np.random.rand(200, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(200)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=42)
 
gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120, random_state=42)
gbrt.fit(X_train, y_train)
 
errors = [mean_squared_error(y_val, y_pred) for y_pred in gbrt.staged_predict(X_val)]
best_n_estimators = np.argmin(errors) + 1
print("best n_estimators:", best_n_estimators)
 
gbrt_best = GradientBoostingRegressor(max_depth=2, n_estimators=best_n_estimators, random_state=42)
gbrt_best.fit(X_train, y_train)
print("validation MSE:", round(mean_squared_error(y_val, gbrt_best.predict(X_val)), 5))
Finding the best number of trees
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
np.random.seed(42)
X = np.random.rand(200, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(200)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=42)
 
gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=120, random_state=42)
gbrt.fit(X_train, y_train)
 
errors = [mean_squared_error(y_val, y_pred) for y_pred in gbrt.staged_predict(X_val)]
best_n_estimators = np.argmin(errors) + 1
print("best n_estimators:", best_n_estimators)
 
gbrt_best = GradientBoostingRegressor(max_depth=2, n_estimators=best_n_estimators, random_state=42)
gbrt_best.fit(X_train, y_train)
print("validation MSE:", round(mean_squared_error(y_val, gbrt_best.predict(X_val)), 5))
text
best n_estimators: 54
validation MSE: 0.00266
text
best n_estimators: 54
validation MSE: 0.00266

Instead of training all 120 trees and looking back, you can also set warm_start=Truewarm_start=True and stop the loop the moment validation error hasn’t improved for a few rounds in a row — true early stopping, not just hindsight. With warm_start=Truewarm_start=True, calling .fit().fit() again doesn’t start over; scikit-learn keeps the trees it already has and simply adds more, so you can grow the ensemble one tree at a time and bail out as soon as things stop improving:

True early stopping with warm_start
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
np.random.seed(42)
X = np.random.rand(200, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(200)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=42)
 
gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True, random_state=42)
 
min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping
 
print("stopped at n_estimators:", n_estimators)
print("best validation MSE:", round(min_val_error, 5))
True early stopping with warm_start
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
 
np.random.seed(42)
X = np.random.rand(200, 1) - 0.5
y = 3 * X[:, 0] ** 2 + 0.05 * np.random.randn(200)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=42)
 
gbrt = GradientBoostingRegressor(max_depth=2, warm_start=True, random_state=42)
 
min_val_error = float("inf")
error_going_up = 0
for n_estimators in range(1, 120):
    gbrt.n_estimators = n_estimators
    gbrt.fit(X_train, y_train)
    y_pred = gbrt.predict(X_val)
    val_error = mean_squared_error(y_val, y_pred)
    if val_error < min_val_error:
        min_val_error = val_error
        error_going_up = 0
    else:
        error_going_up += 1
        if error_going_up == 5:
            break  # early stopping
 
print("stopped at n_estimators:", n_estimators)
print("best validation MSE:", round(min_val_error, 5))
text
stopped at n_estimators: 49
best validation MSE: 0.00272
text
stopped at n_estimators: 49
best validation MSE: 0.00272

Five rounds in a row without improvement was enough to know 49 trees was close enough to optimal — no need to train the full 120 and look back, and no wasted trees past the point of diminishing returns.

Scikit-learn’s fast baseline: HistGradientBoosting

For classification (or when your dataset is larger), scikit-learn’s histogram-based boosters are a fast, strong default — inspired by the same binning trick LightGBM made popular:

HistGradientBoostingClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
hgb = HistGradientBoostingClassifier(random_state=42)
hgb.fit(X_train, y_train)
print(round(accuracy_score(y_test, hgb.predict(X_test)), 3))
HistGradientBoostingClassifier
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
 
X, y = make_moons(n_samples=500, noise=0.30, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
 
hgb = HistGradientBoostingClassifier(random_state=42)
hgb.fit(X_train, y_train)
print(round(accuracy_score(y_test, hgb.predict(X_test)), 3))
text
0.896
text
0.896

XGBoost / LightGBM / CatBoost (high level)

An optimized, hugely popular implementation of Gradient Boosting is available in the xgboostxgboost library — Extreme Gradient Boosting — and its API mirrors scikit-learn’s:

XGBoost API (mirrors scikit-learn)
import xgboost
 
xgb_reg = xgboost.XGBRegressor()
xgb_reg.fit(X_train, y_train)
y_pred = xgb_reg.predict(X_val)
 
# built-in early stopping
xgb_reg.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=2,
)
XGBoost API (mirrors scikit-learn)
import xgboost
 
xgb_reg = xgboost.XGBRegressor()
xgb_reg.fit(X_train, y_train)
y_pred = xgb_reg.predict(X_val)
 
# built-in early stopping
xgb_reg.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    early_stopping_rounds=2,
)
  • XGBoost — battle-tested, strong defaults, a fixture of ML competition leaderboards
  • LightGBM — leaf-wise growth and histogram binning make it very fast on large data
  • CatBoost — built-in handling of categorical features, less preprocessing needed

These external libraries are commonly used in industry, but they’re optional here — the concepts above (residual fitting, shrinkage, early stopping) apply to all of them.

Visualize it: residuals shrink stage by stage

Each new stage refines the previous approximation of the true curve. Watch the blue step-function ensemble hug the amber target curve more tightly as more trees join — and watch the red residual segments (what’s left over for the next tree to fit) shrink toward nothing as the residual readout at the top right ticks down.

sketch Sequential residual fitting p5.js
More trees (stages) let the ensemble's step function approximate the true curve more closely, while the red residual gaps and the residual readout shrink toward zero.

🧪 Try It Yourself

Exercise 1 – Fit HistGradientBoostingClassifier

Exercise 2 – Compute the Residual by Hand

Exercise 3 – Find the Best Number of Trees

Next

Continue to Stacking and Voting Classifiers — instead of reweighting instances or fitting residuals, learn how to combine entirely different model families, either by a simple vote or by training a “meta-learner” on top of them.

If this helped you, consider buying me a coffee ☕

Buy me a coffee

Was this page helpful?

Let us know how we did