Skip to content

Tree-Based Regression

Linear models are powerful when the relationship between features and target is roughly linear — but the real world rarely cooperates. House prices do not rise linearly with square footage forever. Customer lifetime value is not a smooth additive combination of user attributes. Tree-based models capture these nonlinear, interaction-heavy relationships by learning a sequence of if-then decision rules, and their ensemble variants are among the most reliable general-purpose regression models available.

Learning Objectives

  • Explain how a decision tree makes a regression prediction using splits and leaf values
  • Understand why a single decision tree overfits and how max_depth controls complexity
  • Describe how Random Forest reduces variance through bagging and averaging
  • Understand the sequential error-correction logic of Gradient Boosting
  • Read feature importances and know their limitations
  • Know when to use each tree-based model and how to tune the key hyperparameters

Decision Tree Regressor — How Splits Work

A decision tree partitions the feature space into rectangular regions by making a series of binary splits. At each node, it asks: "Which feature and threshold best separates this group into two groups with lower internal variance?"

For regression, "lower internal variance" means the split that produces the greatest reduction in the sum of squared errors within each child node (also called variance reduction or information gain for regression).

Once the tree is built, every leaf node contains a set of training examples. When a new example falls into a leaf, the prediction is the mean of the training targets in that leaf.

Info

The key difference between classification and regression trees: Classification trees assign the majority class label at each leaf. Regression trees assign the mean target value. The splitting criterion is also different — regression trees split to minimise within-leaf variance rather than to maximise class purity (Gini/entropy).

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.tree import DecisionTreeRegressor, export_text
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score

housing = fetch_california_housing(as_frame=True)
X, y = housing.data, housing.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Shallow tree — interpretable
shallow_tree = DecisionTreeRegressor(max_depth=3, random_state=42)
shallow_tree.fit(X_train, y_train)

print("Shallow Decision Tree (max_depth=3):")
print(export_text(shallow_tree, feature_names=list(X.columns)))
# Output (first few lines):
# |--- MedInc <= 5.03
# |   |--- Latitude <= 34.05
# |   |   |--- MedInc <= 3.04
# |   |   |   |--- value: [1.17]
# |   |   |--- MedInc > 3.04
# |   |   |   |--- value: [1.86]
# ...

A depth-3 tree produces at most 8 leaf nodes. Every prediction for a new house will fall into one of those 8 buckets, each with its own mean price. You can read the logic: "If median income ≤ 5.03 AND latitude ≤ 34.05 AND median income ≤ 3.04, predict $117,000."

The Overfitting Problem with Deep Trees

# Compare shallow vs. deep vs. unlimited trees
depths = [2, 3, 5, 10, None]
results = []

for depth in depths:
    tree = DecisionTreeRegressor(max_depth=depth, random_state=42)
    tree.fit(X_train, y_train)

    train_r2 = r2_score(y_train, tree.predict(X_train))
    test_r2  = r2_score(y_test, tree.predict(X_test))
    test_mae = mean_absolute_error(y_test, tree.predict(X_test))

    results.append({
        "max_depth": str(depth),
        "Train R²": round(train_r2, 3),
        "Test R²":  round(test_r2, 3),
        "Test MAE": round(test_mae, 3),
        "Leaves": tree.get_n_leaves()
    })

comparison = pd.DataFrame(results).set_index("max_depth")
print(comparison)
# Output:
#            Train R²  Test R²  Test MAE  Leaves
# max_depth
# 2             0.490    0.484     0.675       4
# 3             0.606    0.579     0.631       8
# 5             0.730    0.641     0.566      32
# 10            0.939    0.672     0.536     693
# None          1.000    0.617     0.550   12204

An unlimited tree achieves perfect training R² (1.000) — it has memorised every training example. But test R² drops back below the depth-10 tree. This is textbook overfitting: the model has learned the noise in the training data as if it were signal.

Warning

A single decision tree almost always overfits if allowed to grow without constraint. The unlimited tree above has 12,204 leaves for 16,512 training examples — it is essentially memorising the dataset. Use max_depth, min_samples_leaf, or min_samples_split to constrain it. Better yet, use a Random Forest instead, which makes overfitting far less of a concern.

Random Forest Regressor — Variance Reduction Through Bagging

A single decision tree is unstable: change a few training examples and the entire tree structure can change. Random Forest solves this by building many trees independently and averaging their predictions. The averaging process reduces variance without increasing bias — the same mechanism that makes the average of many noisy measurements more accurate than any single measurement.

Two sources of randomness in Random Forest: 1. Bootstrap sampling (bagging): Each tree is trained on a random sample of the training data (with replacement). Roughly 63% of examples appear in each sample; the rest are the "out-of-bag" set. 2. Random feature subsets: At each split, each tree considers only a random subset of max_features features (default: all features for regression). This decorrelates the trees — if one feature dominates, not every tree gets to use it at every split.

from sklearn.ensemble import RandomForestRegressor

rf = RandomForestRegressor(
    n_estimators=200,      # number of trees
    max_features="sqrt",   # features per split — "sqrt" is a common heuristic
    max_depth=None,        # individual trees can be deep; averaging saves us
    min_samples_leaf=4,    # each leaf must have at least 4 examples — slight regularisation
    n_jobs=-1,             # use all CPU cores
    random_state=42
)
rf.fit(X_train, y_train)
y_pred_rf = rf.predict(X_test)

print(f"Random Forest (200 trees)")
print(f"  MAE: {mean_absolute_error(y_test, y_pred_rf):.3f}")
print(f"  R²:  {r2_score(y_test, y_pred_rf):.3f}")
# Output:
# Random Forest (200 trees)
#   MAE: 0.328
#   R²:  0.805

A substantial jump from the linear model's R² of 0.576 to 0.805 — the nonlinear, interaction-based relationships in house prices are being captured.

Out-of-Bag Score

Because each tree is trained on roughly 63% of data, the remaining 37% can be used as a built-in validation set. This is the out-of-bag (OOB) score — a free estimate of generalisation performance without needing a separate validation set.

rf_with_oob = RandomForestRegressor(
    n_estimators=200, oob_score=True, n_jobs=-1, random_state=42
)
rf_with_oob.fit(X_train, y_train)

print(f"OOB R² score: {rf_with_oob.oob_score_:.3f}")
# Output:
# OOB R² score: 0.798
# (Slightly lower than test R² — both are reasonable estimates of true generalisation)

Feature Importance from Random Forest

feature_importances = pd.Series(
    rf.feature_importances_,
    index=X.columns
).sort_values(ascending=False)

print("Feature importances:")
print(feature_importances.round(3))
# Output:
# MedInc        0.524
# AveOccup      0.140
# Latitude      0.086
# Longitude     0.086
# HouseAge      0.054
# AveRooms      0.043
# Population    0.036
# AveBedrms     0.031

# Visualise
feature_importances.plot(kind="barh", figsize=(8, 5))
plt.xlabel("Feature Importance (mean decrease in impurity)")
plt.title("Random Forest Feature Importances — California Housing")
plt.gca().invert_yaxis()
plt.tight_layout()
plt.show()

Warning

Feature importance from tree models is impurity-based (MDI) and has known biases. High-cardinality features (many unique values) and continuous features are systematically ranked higher than binary features, even when the binary feature is genuinely more important. For more reliable importance, use permutation importance (sklearn.inspection.permutation_importance) or SHAP values. Use MDI importance for quick exploration, not final conclusions.

Gradient Boosting — Sequential Error Correction

Random Forest builds trees independently and averages. Gradient Boosting builds trees sequentially, where each new tree is trained to predict the residual errors of the ensemble so far. It literally corrects its own mistakes in stages.

The algorithm at a conceptual level:

  1. Start with a simple prediction (the mean of y)
  2. Compute the residuals (errors) of this prediction
  3. Fit a small tree to predict those residuals
  4. Add the tree's predictions (scaled by learning_rate) to the current ensemble
  5. Compute new residuals
  6. Repeat for n_estimators iterations

Each tree targets what the current ensemble is getting wrong. The ensemble's predictions converge toward the true values step by step.

Info

The name "Gradient Boosting" comes from the fact that each new tree is fitted to the negative gradient of the loss function with respect to the current predictions. For squared error loss, this gradient is exactly the residuals — which is why residual-fitting is the intuitive way to describe it.

from sklearn.ensemble import GradientBoostingRegressor

gb = GradientBoostingRegressor(
    n_estimators=300,     # number of boosting stages (trees)
    learning_rate=0.05,   # shrinkage per step — smaller = more conservative
    max_depth=4,          # depth of each individual tree
    subsample=0.8,        # fraction of training data per tree (stochastic boosting)
    random_state=42
)
gb.fit(X_train, y_train)
y_pred_gb = gb.predict(X_test)

print(f"Gradient Boosting (300 trees, lr=0.05)")
print(f"  MAE: {mean_absolute_error(y_test, y_pred_gb):.3f}")
print(f"  R²:  {r2_score(y_test, y_pred_gb):.3f}")
# Output:
# Gradient Boosting (300 trees, lr=0.05)
#   MAE: 0.299
#   R²:  0.843

The learning_rate / n_estimators Trade-off

These two hyperparameters interact directly. Lower learning_rate means each tree contributes less, which means you need more trees to reach the same performance. But more trees at a lower learning rate often generalise better:

lr_n_estimators_configs = [
    {"learning_rate": 0.3,  "n_estimators": 100},
    {"learning_rate": 0.1,  "n_estimators": 200},
    {"learning_rate": 0.05, "n_estimators": 400},
    {"learning_rate": 0.01, "n_estimators": 1000},
]

for config in lr_n_estimators_configs:
    model = GradientBoostingRegressor(
        max_depth=4, subsample=0.8, random_state=42, **config
    )
    model.fit(X_train, y_train)
    r2_val = r2_score(y_test, model.predict(X_test))
    print(f"lr={config['learning_rate']:.2f}, n={config['n_estimators']:>4}: R²={r2_val:.3f}")

# Output:
# lr=0.30, n= 100: R²=0.831
# lr=0.10, n= 200: R²=0.843
# lr=0.05, n= 400: R²=0.849
# lr=0.01, n=1000: R²=0.837

Tip

A good starting strategy for Gradient Boosting: set learning_rate=0.05 and n_estimators=1000, then use early stopping to find the right number of trees. This prevents both underfitting (too few trees) and overfitting (too many). sklearn's GradientBoostingRegressor supports n_iter_no_change for early stopping.

XGBoost — The Competition-Grade Booster

XGBoost (eXtreme Gradient Boosting) is the Gradient Boosting algorithm with a more sophisticated regularised objective, faster computation, built-in handling of missing values, and superior out-of-the-box performance. It won hundreds of Kaggle competitions and remains one of the most reliable choices for tabular regression.

# pip install xgboost
from xgboost import XGBRegressor

xgb = XGBRegressor(
    n_estimators=500,
    learning_rate=0.05,
    max_depth=5,
    subsample=0.8,
    colsample_bytree=0.8,  # fraction of features per tree
    reg_alpha=0.1,         # L1 regularisation on weights
    reg_lambda=1.0,        # L2 regularisation on weights
    random_state=42,
    n_jobs=-1,
    verbosity=0
)
xgb.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)
y_pred_xgb = xgb.predict(X_test)

print(f"XGBoost")
print(f"  MAE: {mean_absolute_error(y_test, y_pred_xgb):.3f}")
print(f"  R²:  {r2_score(y_test, y_pred_xgb):.3f}")
# Output:
# XGBoost
#   MAE: 0.293
#   R²:  0.853

Model Comparison: All Regressors

from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

models_to_compare = {
    "Linear Regression": Pipeline([
        ("scaler", StandardScaler()),
        ("model", LinearRegression())
    ]),
    "Ridge CV": Pipeline([
        ("scaler", StandardScaler()),
        ("model", RidgeCV(alphas=np.logspace(-3, 4, 50)))
    ]),
    "Decision Tree (depth=5)": DecisionTreeRegressor(max_depth=5, random_state=42),
    "Random Forest (200)":     RandomForestRegressor(n_estimators=200, n_jobs=-1, random_state=42),
    "Gradient Boosting":       GradientBoostingRegressor(n_estimators=300, learning_rate=0.05,
                                                          max_depth=4, random_state=42),
}

print(f"{'Model':<30} {'MAE':>7} {'R²':>7}")
print("-" * 46)
for name, model in models_to_compare.items():
    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    print(f"{name:<30} {mean_absolute_error(y_test, preds):>7.3f} {r2_score(y_test, preds):>7.3f}")

# Output:
# Model                          MAE       R²
# ----------------------------------------------
# Linear Regression             0.533    0.576
# Ridge CV                      0.533    0.576
# Decision Tree (depth=5)       0.566    0.641
# Random Forest (200)           0.328    0.805
# Gradient Boosting             0.299    0.843

Success

Choosing between Random Forest and Gradient Boosting in practice:

  • Random Forest: Faster to train, more robust to hyperparameters, harder to overfit catastrophically. Good default choice.
  • Gradient Boosting / XGBoost: Higher ceiling. With careful tuning, it usually outperforms Random Forest. Requires more care with learning_rate, n_estimators, and early stopping.
  • Rule of thumb: Start with Random Forest. If you need better performance and have time to tune, move to XGBoost or LightGBM.

Tree Models Do Not Need Feature Scaling

Unlike linear models, tree models split on thresholds — they are indifferent to the scale of the features. A feature measured in dollars and a feature measured in percentages will both be split at their respective thresholds without any scale distortion. You do not need StandardScaler for tree-based models.

# Both of these produce identical predictions
rf_unscaled = RandomForestRegressor(n_estimators=100, random_state=42)
rf_unscaled.fit(X_train, y_train)  # no scaling — correct

# Scaling does not hurt, but it also does not help for trees
# Do not add a scaler to tree model pipelines unless you have a specific reason

Tip

This scale-invariance is one reason tree-based models are popular in practice — they work well straight out of the box on heterogeneous tabular data where features are on wildly different scales.

What's Next

You've covered decision trees and their depth-overfitting tradeoff, Random Forests and bagging, Gradient Boosting with the learning_rate/n_estimators interaction, XGBoost for competition-grade performance, a full model comparison on one dataset, and why tree models do not need feature scaling. Next up: 05-regression-metrics — where you'll learn how to choose between MAE, RMSE, R², and MAPE, understand what each metric hides and reveals, and build the diagnostic plots that reveal where your model systematically fails.

Optional Deep Dive

Read the XGBoost paper "XGBoost: A Scalable Tree Boosting System" by Chen and Guestrin (2016, available at arxiv.org/abs/1603.02754) — it explains the regularised objective function, the approximate split-finding algorithm, and the system-level optimisations that make XGBoost faster and more accurate than sklearn's GradientBoostingRegressor.

03-ridge-lasso-elasticnet | 05-regression-metrics