Regression Trees

A regression tree guesses a number by asking yes-or-no questions about the features and answering with the average label of the training rows that end in the same leaf, so its guesses are steps, never outside the range of the training labels, and a tree with no depth limit memorizes its training rows.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Train a DecisionTreeRegressor with max_depth and random_state, and read its yes-or-no questions and the averages its leaves guess
  • Explain why a regression tree's guesses are steps that stay inside its training labels' range, and why a deep tree scores perfectly in training

Key takeaways

  • A regression tree asks yes-or-no questions, such as "rainfall at most 16.80 inches?", and its guess is the average label of the training rows that ended in the same leaf.
  • DecisionTreeRegressor(max_depth=3, random_state=0) limits the tree to three questions deep and fixes its tie-breaking, so every run builds the same tree.
  • A tree's guesses are flat steps, not a smooth curve, and they never go below the lowest or above the highest label it trained on.
  • A tree with no depth limit can fit its training rows perfectly and still miss new rows by much more, the first sign of overfitting.

Curves with Polynomial Features followed the orchard's hump, but only because you guessed its shape: rain squared. Real data rarely tells you its shape up front, and a person asked to guess a harvest wouldn't reach for a formula anyway. They'd ask questions: "Was it a dry year? Very dry?" Then they'd remember how trees did in years like that. Trees of questions show up all over machine learning, alone or in groups (Random Forests comes later in the course), and they start here.

The example is Hilltop Orchard's orchard.csv, from Exercise: Orchard Harvest Models. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

Questions that end in an average

A regression tree is a set of yes-or-no questions about the features, one after another, where each final answer is a number. scikit-learn's is DecisionTreeRegressor (decision tree is the general name; regression says it guesses a number). export_text prints the questions it learned:

import pandas as pd
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor, export_text

orchard = pd.read_csv("orchard.csv")
X = orchard[["rainfall_inches"]]
y = orchard["apples_lbs"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

tree = DecisionTreeRegressor(max_depth=3, random_state=0)
tree.fit(X_train, y_train)
print(export_text(tree, feature_names=["rainfall_inches"]))
|--- rainfall_inches <= 16.80
|   |--- rainfall_inches <= 11.47
|   |   |--- rainfall_inches <= 10.21
|   |   |   |--- value: [118.00]
|   |   |--- rainfall_inches >  10.21
|   |   |   |--- value: [165.25]
|   |--- rainfall_inches >  11.47
|   |   |--- rainfall_inches <= 14.47
|   |   |   |--- value: [208.21]
|   |   |--- rainfall_inches >  14.47
|   |   |   |--- value: [226.89]
|--- rainfall_inches >  16.80
|   |--- rainfall_inches <= 43.38
|   |   |--- rainfall_inches <= 19.63
|   |   |   |--- value: [271.33]
|   |   |--- rainfall_inches >  19.63
|   |   |   |--- value: [314.34]
|   |--- rainfall_inches >  43.38
|   |   |--- rainfall_inches <= 47.28
|   |   |   |--- value: [221.55]
|   |   |--- rainfall_inches >  47.28
|   |   |   |--- value: [173.00]

Read it from the top. <= is read "less than or equal to": the first question is "rainfall at most 16.80 inches?" A tree with 30 inches answers no, then yes (at most 43.38), then no (more than 19.63), and lands on value: [314.34]. Each place a path ends is a leaf, and its value is the tree's guess.

Where does 314.34 come from? It's the average harvest of the training trees that landed in that leaf:

middle = (X_train["rainfall_inches"] > 19.63) & (X_train["rainfall_inches"] <= 43.38)
print(middle.sum(), "training trees, average", round(y_train[middle].mean(), 2))
85 training trees, average 314.34

fit chose each question, one at a time, to split the training trees into groups whose harvests are as alike as possible (the smallest squared errors around each group's average), and each leaf keeps its group's average.

max_depth and random_state

max_depth=3 allows at most three questions on any path, so this tree has 8 leaves. Leave it out, and the tree keeps asking until every leaf is pure or too small to split. scikit-learn's own advice is to start at 3, look at the tree, then go deeper.

random_state=0 fixes the tree's tie-breaking. While it builds, the tree looks at the features in a shuffled order, and when two questions split equally well it has to pick one; with a fixed random_state, every run builds the same tree, as with the split's seed.

print("Test MAE:", round(mean_absolute_error(y_test, tree.predict(X_test)), 1))
Test MAE: 26.4

Off by about 26 pounds on the test trees: far better than the straight line's 54.8, not as close as the curve's 17.5.

Steps, not curves

Ask the tree about rain from 15 to 48 inches:

rains = pd.DataFrame({"rainfall_inches": [15, 18, 25, 30, 35, 45, 48]})
for rain, guess in zip(rains["rainfall_inches"], tree.predict(rains)):
    print(rain, "inches:", round(guess, 2))
15 inches: 226.89
18 inches: 271.33
25 inches: 314.34
30 inches: 314.34
35 inches: 314.34
45 inches: 221.55
48 inches: 173.0

25, 30 and 35 inches get the same guess: they land in the same leaf. A tree's guesses are flat steps that jump at each question's number. A deeper tree has more, smaller steps, but never a smooth curve.

The same 160 training trees as dots, rainfall across the bottom and pounds of apples up the side, with the depth-3 tree's guesses drawn as eight flat steps: low steps on the dry side, at 118, 165.25, 208.21 and 226.89 pounds, then 271.33, one long flat step at 314.34 pounds from 19.63 to 43.38 inches across the top of the hump, then 221.55 and 173 pounds on the wet side

And the steps never leave the range of the training harvests. Every leaf is an average of training labels, so no guess can be lower than the smallest or higher than the largest:

outside = pd.DataFrame({"rainfall_inches": [2, 10, 49, 70]})
print(tree.predict(outside).round(2))
print("Training harvests:", y_train.min(), "to", y_train.max())
[118. 118. 173. 173.]
Training harvests: 111 to 385

Two inches of rain gets the same guess as 10; seventy inches, the same as 49. Beyond the data, a tree keeps the nearest leaf's answer: no wild swings like a high-degree curve, but no new information either. scikit-learn's documentation says it plainly: trees aren't good at extrapolation, guessing beyond the data they learned from.

A tree that remembers every tree

What if the tree can ask as many questions as it likes? Give it the tree's age too, and no depth limit:

both = ["rainfall_inches", "tree_age_years"]
X_train, X_test, y_train, y_test = train_test_split(orchard[both], y, test_size=0.2, random_state=42)
deep = DecisionTreeRegressor(random_state=0)
deep.fit(X_train, y_train)
print("Depth:", deep.get_depth(), "Leaves:", deep.get_n_leaves())
print("Training MAE:", round(mean_absolute_error(y_train, deep.predict(X_train)), 1))
print("Test MAE:", round(mean_absolute_error(y_test, deep.predict(X_test)), 1))
Depth: 19 Leaves: 155
Training MAE: 0.0
Test MAE: 24.6

155 leaves for 160 training trees: nearly one leaf per tree. On the training trees it's never wrong, a perfect score. On the test trees it's off by about 25 pounds. It memorized each tree's harvest, random wobble and all, instead of the pattern, like a student who memorized last year's answer key. That's called overfitting, and Overfitting and Underfitting in Module 4 explains it in full. For now, the lesson is this: a training score alone can't tell you a model is good.

Check yourself

2 questions to try on the lesson page.

So what

A regression tree finds the shape of the data by itself, one question at a time, but its guesses come in steps, stop at the edges of what it saw, and turn into memorizing when it's allowed to grow without a limit.

Sources

  • scikit-learn, Decision Trees, User Guide: trees can be applied to regression with DecisionTreeRegressor; their predictions are piecewise constant, neither smooth nor continuous, so they aren't good at extrapolation; over-complex trees that don't generalize are overfitting, which setting the maximum depth helps avoid; with the mean squared error, a leaf's value is the mean of the training targets in it; and the tip to start with max_depth=3 and then increase the depth.
  • scikit-learn 1.9.1, DecisionTreeRegressor, API reference: the default criterion, "squared_error", uses the mean of each leaf; max_depth=None expands nodes until all leaves are pure or too small to split; the features are randomly permuted at each split, so random_state must be fixed to an integer for the same tree every run; get_depth and get_n_leaves.
  • scikit-learn 1.9.1, export_text, API reference: builds a text report showing the rules of a decision tree.

Key terms

Related