Regression Metrics¶
A model that looks great by one metric can be quietly failing by another. The practitioner who only checks R² is the practitioner who ships a model with perfect average performance but catastrophic failures on high-value predictions. Understanding what each metric actually measures — and what it hides — is not a formality. It is how you avoid being wrong with confidence.
Learning Objectives¶
- Calculate MAE, MSE, RMSE, R², Adjusted R², and MAPE and explain what each number means
- Identify which metric fits which business problem
- Understand when R² is misleading and what to look for instead
- Read a residual plot and identify patterns that indicate model failure
- Build a complete diagnostic workflow: metrics + residuals + distribution check
What a Metric Is Measuring¶
Every regression metric answers one question: how far are the model's predictions from the true values? They differ in how they measure that distance and what kinds of errors they emphasise.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import (
mean_absolute_error,
mean_squared_error,
r2_score
)
housing = fetch_california_housing(as_frame=True)
X, y = housing.data, housing.target
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
model = RandomForestRegressor(n_estimators=200, n_jobs=-1, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
residuals = y_test.values - y_pred
MAE — Mean Absolute Error¶
MAE is the average absolute difference between prediction and truth. It treats every unit of error equally — a mistake of 2 is exactly twice as bad as a mistake of 1.
mae = mean_absolute_error(y_test, y_pred)
print(f"MAE: {mae:.3f}")
print(f" → On average, predictions are off by ${mae * 100_000:,.0f}")
# Output:
# MAE: 0.328
# → On average, predictions are off by $32,800
When to use MAE:
- When the business cares about average accuracy equally across all predictions
- When outliers exist in the target but you do not want them to dominate the metric
- When you need to communicate the metric to a non-technical audience — "off by $32,800 on average" is immediately understandable
Tip
MAE is the metric to report to business stakeholders. "Our model predicts house prices with an average error of $32,800" is concrete, interpretable, and directly tied to the cost of being wrong.
MSE — Mean Squared Error¶
MSE squares the errors before averaging. A prediction error of 10 contributes 100 to MSE. An error of 100 contributes 10,000. Large errors are penalised disproportionately.
The number 0.252 is in squared units of $100k² — not directly interpretable. MSE is the standard training loss for regression (it is what the optimiser minimises), but RMSE is the metric you report.
When to use MSE:
- As the training objective (the algorithm minimises this internally)
- When comparing models during development and you want to penalise large errors heavily
RMSE — Root Mean Squared Error¶
RMSE brings the metric back to the same units as the target, while still penalising large errors more than MAE does.
rmse = np.sqrt(mse) # or: mean_squared_error(y_test, y_pred, squared=False)
print(f"RMSE: {rmse:.3f}")
print(f" → In original units: ${rmse * 100_000:,.0f}")
# Output:
# RMSE: 0.502
# → In original units: $50,200
RMSE of $50,200 vs. MAE of $32,800. The gap tells a story: RMSE > MAE means the model is making some large individual errors that inflate the squared average. Those expensive outlier predictions are worth investigating.
When to use RMSE:
- When large prediction errors are especially costly (a model that is off by $200,000 on one house is not equivalent to being off by $10,000 on 20 houses)
- When you want a metric in the same units as the target that also penalises outlier predictions
- When the distribution of errors is approximately normal (RMSE is most interpretable in that case)
Warning
RMSE is sensitive to outlier predictions. A single prediction error of 5 while all others are 0.2 will dominate the RMSE. If your dataset has a few extremely high-value targets (luxury homes, enterprise contracts), RMSE may tell you your model is failing when it is actually performing well on the bulk of cases. Check whether the high RMSE is driven by a small cluster of examples.
R² — Coefficient of Determination¶
R² answers: "How much of the variance in the target does the model explain?" A model that always predicts the mean of y has R² = 0. A perfect model has R² = 1. A model worse than always predicting the mean has R² < 0.
r2 = r2_score(y_test, y_pred)
print(f"R²: {r2:.3f}")
print(f" → The model explains {r2*100:.1f}% of the variance in house prices")
# Output:
# R²: 0.805
# → The model explains 80.5% of the variance in house prices
When to use R²:
- When comparing models trained on the same dataset — R² provides a normalised scale
- When communicating model performance to technical audiences who understand the variance-explained framing
- As one metric among several — never as the sole evaluation metric
Warning
R² never decreases when you add features, even irrelevant ones. This is because adding a feature always gives the model more flexibility to fit the training data, even if it is fitting noise. An R² of 0.95 computed on the training set is almost meaningless. Always evaluate R² on held-out test data.
The R² Trap — When a Good Score Hides a Bad Model¶
Consider these two models, both with R² = 0.82:
# Illustration: same R², very different residual patterns
np.random.seed(42)
x_demo = np.linspace(0, 10, 200)
y_true_demo = 3 * x_demo + np.random.normal(0, 2, 200)
# Model A: good fit
y_pred_good = 3 * x_demo + np.random.normal(0, 2.1, 200)
# Model B: systematic curve missed (R² looks similar but residuals have pattern)
y_pred_curved = 3 * x_demo - 0.3 * (x_demo - 5)**2 + 5 + np.random.normal(0, 2.1, 200)
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
for ax, y_pred_demo, title in zip(
axes,
[y_pred_good, y_pred_curved],
["Model A: Random residuals (good)", "Model B: Curved residuals (linear term missed)"]
):
residuals_demo = y_true_demo - y_pred_demo
ax.scatter(y_pred_demo, residuals_demo, alpha=0.4, s=15)
ax.axhline(0, color="red", linewidth=1)
ax.set_xlabel("Predicted")
ax.set_ylabel("Residual")
ax.set_title(title)
plt.tight_layout()
plt.show()
# Both models may have similar R² — but Model B has a systematic curve
# in its residuals that reveals a missed nonlinear relationship
Warning
High R² does not mean the model is correct. It means the model explains a large fraction of the variance. If the residuals show a curve, a fan, or clusters, the model is systematically wrong somewhere — and R² may not show you this at all. The residuals vs. fitted plot is non-negotiable.
Adjusted R² — Penalising Extra Features¶
Where n is the number of observations and p is the number of features. Adjusted R² decreases when an added feature does not improve the model enough to justify the added complexity. It is the correct metric to use when comparing models with different numbers of features.
def adjusted_r2(y_true, y_pred, n_features):
r2 = r2_score(y_true, y_pred)
n = len(y_true)
return 1 - (1 - r2) * (n - 1) / (n - n_features - 1)
adj_r2 = adjusted_r2(y_test, y_pred, n_features=X.shape[1])
print(f"R²: {r2:.4f}")
print(f"Adjusted R²: {adj_r2:.4f}")
# Output:
# R²: 0.8054
# Adjusted R²: 0.8051
# (Small difference with 8 features — difference grows with many features)
MAPE — Mean Absolute Percentage Error¶
MAPE expresses error as a percentage of the true value. A MAPE of 12% means predictions are off by 12% on average. This is interpretable in business terms — "our revenue forecast is off by 12% on average."
def mean_absolute_percentage_error(y_true, y_pred):
y_true = np.array(y_true)
y_pred = np.array(y_pred)
return np.mean(np.abs((y_true - y_pred) / y_true)) * 100
mape = mean_absolute_percentage_error(y_test, y_pred)
print(f"MAPE: {mape:.1f}%")
# Output:
# MAPE: 18.2%
Warning
MAPE is undefined when true values are zero and becomes extremely large when true values are near zero. It also penalises underpredictions more than overpredictions of the same magnitude. For a dataset with a wide range of target values (e.g., product prices from $1 to $10,000), MAPE can be dominated by errors on the cheapest items. Use MAPE when the target has a minimum value well above zero and errors are best understood as percentages.
Metric Selection Guide¶
| Situation | Best Metric |
|---|---|
| Communicating to non-technical stakeholders | MAE |
| Large errors are disproportionately costly | RMSE |
| Comparing models trained on the same dataset | R² or RMSE |
| Comparing models with different numbers of features | Adjusted R² |
| Errors matter as a percentage of the true value | MAPE |
| Target spans several orders of magnitude | MAPE or log-transformed MAE |
| Model debugging and diagnosis | Residual plots |
Success
Report multiple metrics. A model report that only says "R² = 0.85" is incomplete. Report at minimum: MAE (interpretable), RMSE (outlier sensitivity), and R² (variance explained). Then show the residual plot. That four-piece summary tells the whole story.
The Residual Plot — Your Most Important Diagnostic Tool¶
The residual plot (residuals vs. fitted values) is the single most informative diagnostic available to a regression analyst. Read it before trusting any metric.
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
# 1. Residuals vs. Fitted
axes[0, 0].scatter(y_pred, residuals, alpha=0.2, s=8, color="steelblue")
axes[0, 0].axhline(0, color="red", linewidth=1.5)
axes[0, 0].set_xlabel("Predicted Value (units of $100k)")
axes[0, 0].set_ylabel("Residual")
axes[0, 0].set_title("Residuals vs. Fitted\n(should look like random cloud)")
# 2. Actual vs. Predicted
axes[0, 1].scatter(y_test, y_pred, alpha=0.2, s=8, color="steelblue")
axes[0, 1].plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()],
color="red", linewidth=1.5, linestyle="--")
axes[0, 1].set_xlabel("Actual Value")
axes[0, 1].set_ylabel("Predicted Value")
axes[0, 1].set_title("Actual vs. Predicted\n(should hug the diagonal)")
# 3. Residual distribution
axes[1, 0].hist(residuals, bins=60, edgecolor="black", color="steelblue")
axes[1, 0].axvline(0, color="red", linewidth=1.5)
axes[1, 0].set_xlabel("Residual")
axes[1, 0].set_title("Residual Distribution\n(ideally bell-shaped, centred at 0)")
# 4. Absolute residuals vs. Fitted (heteroscedasticity check)
axes[1, 1].scatter(y_pred, np.abs(residuals), alpha=0.2, s=8, color="steelblue")
axes[1, 1].set_xlabel("Predicted Value")
axes[1, 1].set_ylabel("|Residual|")
axes[1, 1].set_title("Scale-Location\n(constant spread = homoscedastic)")
plt.suptitle("Regression Diagnostic Plots — Random Forest on California Housing",
fontsize=13, y=1.01)
plt.tight_layout()
plt.show()
Reading the Plots¶
| Plot | What You Want to See | Red Flag |
|---|---|---|
| Residuals vs. Fitted | Random cloud centred on 0 | Curve, fan, or cluster pattern |
| Actual vs. Predicted | Points tight around the diagonal | Systematic above/below-diagonal bias |
| Residual Distribution | Bell-shaped, centred at 0 | Heavy tails, skew, bimodal |
| Scale-Location | Horizontal band of constant width | Widening spread (heteroscedasticity) |
Diagnosing Specific Residual Patterns¶
# Pattern 1: Underestimating high-value predictions (common in house prices)
high_value_mask = y_test > 4.0 # houses > $400k
high_value_residuals = residuals[high_value_mask.values]
print(f"Mean residual for high-value houses: {high_value_residuals.mean():.3f}")
# Negative mean → model systematically underestimates expensive houses
# Pattern 2: Identify the worst predictions
worst_idx = np.argsort(np.abs(residuals))[-10:]
worst_preds = pd.DataFrame({
"Actual": y_test.values[worst_idx],
"Predicted": y_pred[worst_idx],
"Residual": residuals[worst_idx]
})
print("\nTop 10 worst predictions:")
print(worst_preds.round(3))
Tip
When you find systematic patterns in residuals — like the model always underestimating expensive houses — that is not a metric problem, it is a model problem. The fix is usually one of: adding features the model is missing, applying a target transformation (log scale for skewed targets), or using a more expressive model family.
Complete Metrics Summary¶
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
def regression_report(y_true, y_pred, model_name="Model", target_unit_scale=1.0):
"""
Print a complete regression evaluation report.
target_unit_scale: multiply metric values to convert to original units (e.g. 100000 for $100k)
"""
y_true = np.array(y_true)
y_pred = np.array(y_pred)
residuals = y_true - y_pred
mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_true, y_pred)
adj_r2 = 1 - (1 - r2) * (len(y_true) - 1) / (len(y_true) - 1 - 1)
# MAPE — only if no zeros in y_true
if np.all(y_true != 0):
mape = np.mean(np.abs(residuals / y_true)) * 100
mape_str = f"{mape:.2f}%"
else:
mape_str = "N/A (zeros in target)"
print(f"\n{'='*50}")
print(f" Regression Report: {model_name}")
print(f"{'='*50}")
print(f" MAE: {mae:.4f} (±{mae * target_unit_scale:,.0f})")
print(f" RMSE: {rmse:.4f} (±{rmse * target_unit_scale:,.0f})")
print(f" R²: {r2:.4f}")
print(f" Adjusted R²: {adj_r2:.4f}")
print(f" MAPE: {mape_str}")
print(f" Residual std: {np.std(residuals):.4f}")
print(f" Residual mean: {np.mean(residuals):.6f} (near 0 = unbiased)")
print(f"{'='*50}\n")
regression_report(y_test, y_pred, model_name="Random Forest", target_unit_scale=100_000)
# Output:
# ==================================================
# Regression Report: Random Forest
# ==================================================
# MAE: 0.3283 (±32,830)
# RMSE: 0.5021 (±50,210)
# R²: 0.8054
# Adjusted R²: 0.8051
# MAPE: 18.23%
# Residual std: 0.5021
# Residual mean: -0.000012 (near 0 = unbiased)
# ==================================================