What you’ll learn

• how a Decision Tree makes a prediction by walking root-to-leaf
• what “impurity” means, and how Gini and entropy each measure it
• the CART algorithm — how scikit-learn actually chooses each split
• why trees are called “white box” models, and where their instability comes from

What a decision tree is

A decision tree predicts by repeatedly asking “if/else” style questions, starting at the root and walking down until it reaches a leaf with a prediction.

  flowchart TD
  R["Root: Feature <= threshold?"] -->|yes| L[Left branch]
  R -->|no| RR[Right branch]
  L --> P1[Prediction]
  RR --> P2[Prediction]

Géron’s book grows one on the iris dataset, using only petal length and petal width:

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
 
iris = load_iris()
X = iris.data[:, 2:]   # petal length, petal width
y = iris.target
 
tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42)
tree_clf.fit(X, y)

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
 
iris = load_iris()
X = iris.data[:, 2:]   # petal length, petal width
y = iris.target
 
tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42)
tree_clf.fit(X, y)

To classify a new flower, you start at the root (depth 0): is petal length below 2.45 cm? If yes, you land straight on a leaf node predicting Iris setosa — no further questions needed, because that region is perfectly pure. If no, you move to the depth-1 node, which asks a second question: is petal width below 1.75 cm? That decides between Iris versicolor and Iris virginica.

A Decision Tree needs almost no data preparation — no scaling, no centering. That’s one of its most convenient qualities.

How trees choose splits: impurity

At every node, the tree looks at every possible split (feature + threshold) and asks: which split makes the two resulting child nodes as “pure” as possible? A node is pure when every instance in it belongs to the same class.

Gini impurity

Gini impurity for node i is:

Gini_i = 1 - Σ (p_i,k)²Gini_i = 1 - Σ (p_i,k)²

where p_i,kp_i,k is the fraction of instances in node i that belong to class k. A pure node (all one class) has Gini = 0Gini = 0.

For example, in the iris tree above, the depth-1 left node (used only by Iris setosa, p = [1, 0, 0]p = [1, 0, 0]) is already pure: Gini = 0Gini = 0. The depth-2 left node (0 setosa, 49 versicolor, 5 virginica, out of 54 instances) has:

Gini = 1 - (0/54)² - (49/54)² - (5/54)² ≈ 0.168Gini = 1 - (0/54)² - (49/54)² - (5/54)² ≈ 0.168

Entropy (information gain)

Entropy comes from information theory, where it measures the average “surprise” or disorder in a message. In a Decision Tree, a node’s entropy is 0 when it contains only one class:

H_i = -Σ p_i,k * log2(p_i,k)H_i = -Σ p_i,k * log2(p_i,k) (only over classes where p_i,k > 0p_i,k > 0)

That same depth-2 left node has:

H = -(49/54)*log2(49/54) - (5/54)*log2(5/54) ≈ 0.445H = -(49/54)*log2(49/54) - (5/54)*log2(5/54) ≈ 0.445

Both formulas measure the same underlying idea — how “mixed” a node is — but on slightly different scales. Most of the time they lead to very similar trees. Gini is marginally faster to compute (no logarithms), so it’s the scikit-learn default. When they do disagree, Gini tends to isolate the single most frequent class into its own branch, while entropy tends to produce slightly more balanced trees.

The CART training algorithm

Scikit-learn trains trees with the CART (Classification and Regression Tree) algorithm. For classification, at each node it searches over every feature and threshold (k, t_k)(k, t_k) and picks the one that minimizes:

J(k, t_k) = (m_left / m) * Gini_left + (m_right / m) * Gini_rightJ(k, t_k) = (m_left / m) * Gini_left + (m_right / m) * Gini_right

— a weighted average of the two children’s impurities, weighted by how many instances land in each side. Once it finds the best split, CART repeats the same search on each child, recursively, until it hits max_depthmax_depth, or can’t find a split that reduces impurity any further, or another stopping condition kicks in.

Overfitting risk

Left unconstrained, a tree will grow until it fits — and likely overfits — the training data almost perfectly, since nothing stops it from creating one leaf per training instance. Trees are described as nonparametric: not because they lack parameters, but because the number of parameters isn’t fixed ahead of time, so the tree is free to hug the training data as closely as it wants.

Common controls (increasing min_*min_* or decreasing max_*max_* all regularize the tree):

• max_depthmax_depth — maximum depth of the tree
• min_samples_splitmin_samples_split — minimum samples a node must have before it can split
• min_samples_leafmin_samples_leaf — minimum samples required in a leaf
• max_leaf_nodesmax_leaf_nodes — caps the total number of leaves

Instability

Decision Trees only ever split perpendicular to an axis, so they’re sensitive to how the data happens to be rotated — the same linearly separable dataset can look effortless to split one way and awkwardly convoluted after a 45° rotation. They’re also sensitive to small changes in the training data: remove just one instance and retrain, and you may get a noticeably different tree, especially since scikit-learn’s implementation is stochastic (it randomly samples which features to evaluate at each node, unless you fix random_staterandom_state). Random Forests (the next phase) fix this instability by averaging over many trees.

Scikit-learn example

from sklearn.tree import DecisionTreeClassifier
 
tree = DecisionTreeClassifier(
    criterion="gini",  # or "entropy" / "log_loss" depending on sklearn version
    max_depth=5,
    random_state=42,
)

from sklearn.tree import DecisionTreeClassifier
 
tree = DecisionTreeClassifier(
    criterion="gini",  # or "entropy" / "log_loss" depending on sklearn version
    max_depth=5,
    random_state=42,
)

Reading a node’s probabilities

Once trained, a leaf node doesn’t just output a class — it stores the ratio of each class among the training instances that land there, so predict_proba()predict_proba() comes for free:

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
 
iris = load_iris()
X = iris.data[:, 2:]
y = iris.target
 
tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42)
tree_clf.fit(X, y)
 
print(tree_clf.predict_proba([[5, 1.5]]).round(3))
# [[0.    0.907 0.093]]  -> 90.7% Iris versicolor
 
print(tree_clf.predict([[5, 1.5]]))
# [1]

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
 
iris = load_iris()
X = iris.data[:, 2:]
y = iris.target
 
tree_clf = DecisionTreeClassifier(max_depth=2, random_state=42)
tree_clf.fit(X, y)
 
print(tree_clf.predict_proba([[5, 1.5]]).round(3))
# [[0.    0.907 0.093]]  -> 90.7% Iris versicolor
 
print(tree_clf.predict([[5, 1.5]]))
# [1]

Visualize it

A decision tree asks a series of yes/no questions, splitting the data at each node.

  flowchart TD
  A["Sunny?"] -->|yes| B["Humidity high?"]
  A -->|no| C["Windy?"]
  B -->|yes| D["Stay in"]
  B -->|no| E["Play"]
  C -->|yes| F["Stay in"]
  C -->|no| G["Play"]

Mini-checkpoint

Train two trees:

• deep tree (no max_depthmax_depth)
• shallow tree (max_depth=3max_depth=3)

Compare train vs validation scores — the deep tree should score much higher on training data and probably worse on data it hasn’t seen.

🧪 Try It Yourself

Exercise 1 – Compute Gini Impurity by Hand

Exercise 2 – Fit a Tree and Read Its Probabilities

Exercise 3 – Compare Depth 2 vs Depth 3

Next

Continue to Naïve Bayes Classifier for a fast, probabilistic alternative that needs no splitting at all.