Regression Evaluation

Every regression metric answers a slightly different question about your model's errors. MAE asks "how far off are we on average?" RMSE asks "are there catastrophic misses?" R² asks "how much of the variation in the target does the model explain?" Choosing the wrong one means optimising for the wrong thing — and you will not know until the model is in production and someone is asking why the forecast was so far off.

Learning Objectives

• Calculate and interpret MAE, MSE, RMSE, R², Adjusted R², and MAPE
• Choose the right metric based on business cost of errors
• Read a residual plot and identify systematic problems
• Compare models fairly using baseline-relative metrics

---

The Metrics and What They Measure

MAE — Mean Absolute Error

MAE is the average of the absolute differences between predictions and actual values. It is in the same units as the target, which makes it immediately interpretable.

Formula: MAE = mean(|y_actual - y_predicted|)

import numpy as np
from sklearn.metrics import mean_absolute_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

housing = fetch_california_housing()
X_housing, y_housing = housing.data, housing.target  # target is median house value in $100k

X_train, X_test, y_train, y_test = train_test_split(
    X_housing, y_housing, test_size=0.2, random_state=42
)

rf_pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("rf", RandomForestRegressor(n_estimators=100, random_state=42))
])
rf_pipeline.fit(X_train, y_train)
y_pred = rf_pipeline.predict(X_test)

mae = mean_absolute_error(y_test, y_pred)
print(f"MAE: {mae:.3f} (${mae * 100_000:,.0f})")
# Output: MAE: 0.329 ($32,900)
# Interpretation: on average, our prediction is off by $32,900

When to use MAE: when all errors cost roughly the same. A forecast that is $10k wrong is five times as bad as one that is $2k wrong — linear, proportional. MAE is also robust to outliers because it does not square the errors.

MSE — Mean Squared Error

MSE squares each error before averaging. This makes large errors much more expensive than small ones.

Formula: MSE = mean((y_actual - y_predicted)²)

from sklearn.metrics import mean_squared_error

mse = mean_squared_error(y_test, y_pred)
print(f"MSE: {mse:.4f}")
# Output: MSE: 0.2551

MSE is hard to interpret on its own because the units are squared. It is most useful as a loss function during training (it is differentiable, which gradient-based methods need) and for comparing models against each other.

When to use MSE: as a training objective, or when comparing models internally. Do not report MSE to stakeholders — the units are unintuitive.

RMSE — Root Mean Squared Error

RMSE takes the square root of MSE, bringing the metric back to the original units of the target.

Formula: RMSE = sqrt(MSE)

rmse = np.sqrt(mse)
print(f"RMSE: {rmse:.3f} (${rmse * 100_000:,.0f})")
# Output: RMSE: 0.505 ($50,500)

Compare MAE ($32,900) to RMSE ($50,500) on the same predictions. RMSE is larger because it penalises the large errors more heavily. The gap between MAE and RMSE tells you something: a large gap means your model has occasional large misses that MAE is hiding.

Tip

If RMSE is substantially larger than MAE, your model is producing some very large errors on specific cases. Investigate which rows have the largest residuals — that gap is telling you where the model breaks down.

When to use RMSE: as the primary reporting metric when large errors are disproportionately costly. It is the most common regression metric for reporting.

R² — Coefficient of Determination

R² measures what fraction of the variance in the target the model explains. It is unitless and always relative to a baseline (predicting the mean).

Formula: R² = 1 - (SS_residual / SS_total) where SS_total = mean((y - y_mean)²)

from sklearn.metrics import r2_score

r2 = r2_score(y_test, y_pred)
print(f"R²: {r2:.3f}")
# Output: R²: 0.805
# Interpretation: the model explains 80.5% of the variance in house prices

Interpreting R²: - R² = 1.0 — perfect predictions - R² = 0.0 — the model does no better than predicting the mean for every row - R² < 0.0 — the model is worse than predicting the mean (this happens; it is a sign of serious problems)

Warning

R² always increases when you add more features to a linear model, even if those features are random noise. Use Adjusted R² when comparing models with different numbers of features. Never trust R² alone for model selection.

Adjusted R²

Adjusted R² penalises adding features that do not contribute meaningfully to the fit.

Formula: Adj R² = 1 - (1 - R²) * (n - 1) / (n - k - 1) where n is sample size and k is number of features.

def adjusted_r2(r_squared, n_samples, n_features):
    return 1 - (1 - r_squared) * (n_samples - 1) / (n_samples - n_features - 1)

n_samples = X_test.shape[0]
n_features = X_test.shape[1]

adj_r2 = adjusted_r2(r2, n_samples, n_features)
print(f"R²:          {r2:.3f}")
print(f"Adjusted R²: {adj_r2:.3f}")
# Output: R²:          0.805
# Output: Adjusted R²: 0.804
# Small difference here — expected with many samples and few features

Info

Adjusted R² is primarily useful for linear models. For tree-based models or neural networks, it is less meaningful — use cross-validated RMSE or MAE for model comparison instead.

MAPE — Mean Absolute Percentage Error

MAPE expresses error as a percentage of the actual value, making it intuitive for business communication: "our forecast is off by 8% on average."

Formula: MAPE = mean(|y_actual - y_predicted| / |y_actual|) * 100

def mean_absolute_percentage_error(y_actual, y_predicted):
    y_actual = np.array(y_actual)
    y_predicted = np.array(y_predicted)
    # Avoid division by zero
    mask = y_actual != 0
    return np.mean(np.abs((y_actual[mask] - y_predicted[mask]) / y_actual[mask])) * 100

mape = mean_absolute_percentage_error(y_test, y_pred)
print(f"MAPE: {mape:.1f}%")
# Output: MAPE: 16.8%
# Interpretation: on average, predictions are 16.8% off from actual values

Warning

MAPE is undefined when any actual value is zero — division by zero. It also gives disproportionate weight to cases where the actual value is very small (a $1 actual with a $0.50 error = 50% MAPE, but a $10,000 actual with a $5,000 error = 50% MAPE — the latter is far more important). Use MAPE cautiously, and never when your target can be zero.

---

Putting It Together — Compare All Metrics

from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression

# Baseline: predict the mean
baseline = DummyRegressor(strategy="mean")
baseline.fit(X_train, y_train)
y_baseline = baseline.predict(X_test)

# Simple linear model
linear_model = Pipeline([
    ("scaler", StandardScaler()),
    ("lr", LinearRegression())
])
linear_model.fit(X_train, y_train)
y_linear = linear_model.predict(X_test)

models = {
    "Baseline (mean)": y_baseline,
    "Linear Regression": y_linear,
    "Random Forest":    y_pred,
}

print(f"{'Model':<25} {'MAE':>8} {'RMSE':>8} {'R²':>8}")
print("-" * 55)
for model_name, predictions in models.items():
    m_mae  = mean_absolute_error(y_test, predictions)
    m_rmse = np.sqrt(mean_squared_error(y_test, predictions))
    m_r2   = r2_score(y_test, predictions)
    print(f"{model_name:<25} {m_mae:>8.3f} {m_rmse:>8.3f} {m_r2:>8.3f}")

# Output:
# Model                      MAE     RMSE       R²
# -------------------------------------------------------
# Baseline (mean)          0.534    0.736    0.000
# Linear Regression        0.530    0.726    0.030   (barely better than baseline on R²)
# Random Forest            0.329    0.505    0.805

Success

Always include the baseline in your comparison table. A model with R² = 0.03 looks like it's doing something — until you see that the baseline has R² = 0.00 and the improvement is 3 percentage points. Context changes everything.

---

Residual Analysis

An aggregate metric tells you how big the errors are on average. A residual plot tells you whether those errors are random or systematic. Random errors are acceptable — they are the irreducible noise in your data. Systematic errors mean the model is missing something.

import matplotlib.pyplot as plt
import numpy as np

residuals = y_test - y_pred

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

# Plot 1: Residuals vs Predicted
axes[0].scatter(y_pred, residuals, alpha=0.3, s=10, color="steelblue")
axes[0].axhline(y=0, color="red", linestyle="--", linewidth=1)
axes[0].set_xlabel("Predicted values")
axes[0].set_ylabel("Residuals")
axes[0].set_title("Residuals vs Predicted")

# Plot 2: Residual distribution
axes[1].hist(residuals, bins=50, color="steelblue", edgecolor="white")
axes[1].axvline(x=0, color="red", linestyle="--", linewidth=1)
axes[1].set_xlabel("Residual")
axes[1].set_ylabel("Count")
axes[1].set_title("Residual Distribution")

# Plot 3: Actual vs Predicted
axes[2].scatter(y_test, y_pred, alpha=0.3, s=10, color="steelblue")
axes[2].plot([y_test.min(), y_test.max()],
             [y_test.min(), y_test.max()],
             color="red", linestyle="--", linewidth=1)
axes[2].set_xlabel("Actual values")
axes[2].set_ylabel("Predicted values")
axes[2].set_title("Actual vs Predicted")

plt.tight_layout()
plt.savefig("residual_plots.png", dpi=150, bbox_inches="tight")
plt.show()

What to look for in the residuals vs predicted plot:

Pattern	What it means
Random scatter around zero	Model is working well — errors are noise
Fan shape (variance increases with predictions)	Heteroscedasticity — consider log-transforming the target
Curved pattern	The relationship is non-linear and your model is not capturing it
Cluster of large errors at high predictions	The model systematically underestimates high-value cases
Non-zero mean residuals	Systematic bias — model consistently over- or under-predicts

Tip

The residual distribution should be approximately symmetric around zero. A strong left or right skew in residuals means the model is systematically wrong in one direction. Check whether a log transformation of the target fixes this.

---

Choosing Your Metric

Situation	Recommended metric
All errors cost equally	MAE
Large errors are especially costly	RMSE
Need to explain to business stakeholders	MAPE (if no zero targets)
Comparing models with different feature counts	Adjusted R²
General model quality assessment	R² + RMSE
Time series forecasting	MAE or MAPE

Info

In practice, report at least two metrics: one that is absolute (MAE or RMSE, in the units of the target) and one that is relative (R² or MAPE). The absolute metric tells the business how far off predictions are. The relative metric tells you how much of the problem the model solves.

---

---

What's Next

You've covered MAE, MSE, RMSE, MAPE, R², and adjusted R² with their formulas and failure modes, the residuals-vs-predicted diagnostic plot for detecting heteroscedasticity and non-linearity, baseline comparison with DummyRegressor, and the metric selection guide for different business contexts. Next up: 04-classification-evaluation — where you'll learn confusion matrices, precision, recall, F1-score, ROC-AUC, PR-AUC, and threshold selection — the evaluation toolkit for every classification problem where the default 0.5 threshold is almost never the right choice.

Optional Deep Dive

Read "Regression Diagnostics: An Introduction" by John Fox (Sage Publications, Quantitative Applications in the Social Sciences series) — it covers the full set of regression diagnostic tools including influence measures, Cook's distance, and partial regression plots that reveal which individual observations are driving your model's errors.

Previous: Cross-Validation | Next: Classification Evaluation