Evaluation and Interpretation¶
Test set MAE is one number. It does not tell you where the model fails, what drives its decisions, or whether the errors are distributed randomly or concentrated in specific periods. This file works through the full evaluation stack: business-facing metrics, actual vs. predicted plots, error decomposition by time segment, feature importance, and residual analysis.
Learning Objectives¶
- Compute MAE, RMSE, and MAPE and explain the difference to a business audience
- Plot actual vs. predicted sales for the full test window
- Identify whether errors concentrate on specific days of week or months
- Read a feature importance chart and connect it to the EDA findings
- Detect systematic patterns in residuals that reveal model blind spots
- Translate model performance into actionable business language
Setup¶
Continue from Model Building. The complete pipeline — dataset generation, feature engineering, train-test split, and model fitting — must run first. Then add these imports.
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
# gbr, X_train, X_test, y_train, y_test, train, test must already be defined
# feature_cols must be defined from the model-building file
gbr_train_pred = gbr.predict(X_train)
gbr_test_pred = gbr.predict(X_test)
Core Metrics — MAE, RMSE, MAPE¶
Three metrics tell different parts of the story.
def compute_mape(y_true, y_pred):
"""Mean Absolute Percentage Error. Excludes rows where y_true == 0."""
y_true = np.array(y_true, dtype=float)
y_pred = np.array(y_pred, dtype=float)
mask = y_true != 0
return np.mean(np.abs((y_true[mask] - y_pred[mask]) / y_true[mask])) * 100
test_mae = mean_absolute_error(y_test, gbr_test_pred)
test_rmse = np.sqrt(mean_squared_error(y_test, gbr_test_pred))
test_mape = compute_mape(y_test, gbr_test_pred)
test_r2 = r2_score(y_test, gbr_test_pred)
print(f"Test MAE: {test_mae:.1f} units")
print(f"Test RMSE: {test_rmse:.1f} units")
print(f"Test MAPE: {test_mape:.2f}%")
print(f"Test R²: {test_r2:.4f}")
What each metric means:
| Metric | Formula | Interpretation |
|---|---|---|
| MAE | mean(|actual - predicted|) | Average absolute error in original units |
| RMSE | sqrt(mean((actual - predicted)²)) | Like MAE but penalises large errors more heavily |
| MAPE | mean(|actual - predicted| / actual) × 100 | Error as a percentage of actual — scale-independent |
| R² | 1 - SS_res / SS_tot | Fraction of variance explained (1.0 is perfect) |
Info
Why MAPE is preferred for business reporting. Telling the operations team "our model is off by 52 units on average" is less useful than "our model is off by 5.7% on average." A 52-unit error on a category that sells 1,000 units per day is good; the same error on a category that sells 120 units per day is poor. MAPE normalises for scale, making it comparable across categories and interpretable by non-technical stakeholders without needing to know the baseline sales volume.
Warning
MAPE is undefined when actual sales are zero and becomes unstable when actual values are very small. For products with intermittent demand (many zero-sales days), use MAE or weighted MAPE instead.
Actual vs. Predicted — Electronics Test Window¶
Plot the full 6-month test period for Electronics to see how closely the model tracks the actual sales line.
elec_test_mask = test["category_Electronics"] == 1
dates_test = test.loc[elec_test_mask, "date"].values
actuals = y_test[elec_test_mask].values
predictions = gbr_test_pred[elec_test_mask.values]
fig, ax = plt.subplots(figsize=(13, 5))
ax.plot(dates_test, actuals, label="Actual Sales", color="#94A3B8", linewidth=1.0, alpha=0.8)
ax.plot(dates_test, predictions, label="Model Forecast", color="#0D9488", linewidth=1.5, linestyle="--")
ax.fill_between(dates_test, actuals, predictions,
where=(actuals > predictions), alpha=0.15, color="#F59E0B", label="Under-forecast")
ax.fill_between(dates_test, actuals, predictions,
where=(actuals <= predictions), alpha=0.15, color="#3B82F6", label="Over-forecast")
ax.set_title("Electronics — Actual vs. Predicted Sales (Jul–Dec 2023)", fontsize=13)
ax.set_xlabel("Date")
ax.set_ylabel("Units Sold")
ax.legend(loc="upper left")
ax.grid(axis="y", linestyle="--", alpha=0.4)
plt.tight_layout()
plt.show()
elec_mae = mean_absolute_error(actuals, predictions)
elec_mape = compute_mape(actuals, predictions)
print(f"Electronics — Test MAE: {elec_mae:.1f} | MAPE: {elec_mape:.2f}%")
Error by Day of Week — Does the Model Fail on Weekends?¶
Compute residuals and group by day of week to check whether the weekend seasonality is well-captured.
residuals_df = pd.DataFrame({
"date": test["date"].values,
"day_of_week": test["day_of_week"].values,
"month": test["month"].values,
"actual": y_test.values,
"predicted": gbr_test_pred,
})
residuals_df["residual"] = residuals_df["actual"] - residuals_df["predicted"]
residuals_df["abs_error"] = residuals_df["residual"].abs()
day_labels = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
error_by_dow = (
residuals_df.groupby("day_of_week")["abs_error"]
.mean()
.reset_index()
)
error_by_dow["day_label"] = [day_labels[i] for i in error_by_dow["day_of_week"]]
fig, ax = plt.subplots(figsize=(8, 4))
bar_colors = ["#0D9488"] * 5 + ["#F59E0B"] * 2
ax.bar(error_by_dow["day_label"], error_by_dow["abs_error"],
color=bar_colors, edgecolor="white")
ax.set_title("Mean Absolute Error by Day of Week", fontsize=12)
ax.set_xlabel("Day of Week (Amber = Weekend)")
ax.set_ylabel("Mean Absolute Error (units)")
ax.grid(axis="y", linestyle="--", alpha=0.4)
plt.tight_layout()
plt.show()
print(error_by_dow[["day_label", "abs_error"]].to_string(index=False))
Info
Weekend errors are ~30% higher than weekday errors. The is_weekend and day_of_week features capture the average weekend dip, but individual weekend days are noisier — shoppers are less predictable on Saturday and Sunday than on structured workdays. This is not a modelling failure; it reflects genuine higher variance in weekend demand.
Error by Month — Does Accuracy Degrade in Q4?¶
Check whether the Q4 spike creates harder-to-forecast periods.
month_labels_short = ["Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
error_by_month = (
residuals_df.groupby("month")["abs_error"]
.mean()
.reset_index()
)
fig, ax = plt.subplots(figsize=(8, 4))
bar_colors = ["#0D9488"] * 4 + ["#F59E0B"] * 2 # Jul-Oct normal, Nov-Dec Q4
ax.bar(month_labels_short, error_by_month["abs_error"],
color=bar_colors, edgecolor="white")
ax.set_title("Mean Absolute Error by Month (Test Period: Jul–Dec 2023)", fontsize=12)
ax.set_xlabel("Month (Amber = Q4 holiday season)")
ax.set_ylabel("Mean Absolute Error (units)")
ax.grid(axis="y", linestyle="--", alpha=0.4)
plt.tight_layout()
plt.show()
print(error_by_month[["month", "abs_error"]].to_string(index=False))
Warning
November and December errors are 60–70% higher than July–September. The annual seasonality pattern is learned from two previous Q4 seasons. But promotion events, new product launches, and supply chain disruptions in Q4 create spikes the model cannot anticipate. This is a structural limitation: without a promotion calendar, the model treats every day as "average for that month" — which is wrong on any day with an actual promotion.
Feature Importance¶
Inspect which features the Gradient Boosting model relies on most.
importance_df = pd.DataFrame({
"feature": feature_cols,
"importance": gbr.feature_importances_,
}).sort_values("importance", ascending=False)
# Plot top 15 features
top15 = importance_df.head(15)
fig, ax = plt.subplots(figsize=(9, 6))
ax.barh(
top15["feature"][::-1],
top15["importance"][::-1],
color="#0D9488", edgecolor="white", alpha=0.85
)
ax.set_xlabel("Feature Importance (Gini-based)", fontsize=11)
ax.set_title("Top 15 Feature Importances — Gradient Boosting", fontsize=12)
ax.grid(axis="x", linestyle="--", alpha=0.4)
plt.tight_layout()
plt.show()
print(importance_df.head(10).to_string(index=False))
feature importance
lag_7 0.3812
rolling_mean_7 0.1947
lag_28 0.0934
rolling_mean_28 0.0821
lag_14 0.0643
lag_1 0.0518
month 0.0411
quarter 0.0278
day_of_week 0.0224
is_weekend 0.0187
Success
lag_7 alone accounts for 38% of the model's decision-making — consistent with its correlation of 0.91 found in EDA. rolling_mean_7 adds another 19%. Together these two features, both capturing recent sales history, explain over half of the model's predictive power. Calendar features like month, quarter, and day_of_week contribute a meaningful but secondary role. Category one-hot features (not shown in top 10) have low importance because lag features already encode category-level scale implicitly — Electronics consistently has higher lags than Home.
Why lag-7 dominates: The weekly seasonality built into the data creates a near-perfect same-day-last-week signal. A Monday in November 2023 looks very similar to a Monday in October 2023. The model learns this and leans heavily on lag_7 as its primary prediction anchor, then uses rolling means to adjust for the overall recent trend level.
Residuals Over Time¶
Plot residuals chronologically to check for systematic error patterns — periods where the model is consistently too high or too low.
fig, ax = plt.subplots(figsize=(13, 4))
ax.scatter(
residuals_df["date"], residuals_df["residual"],
s=3, alpha=0.4, color="#0D9488"
)
ax.axhline(0, color="#94A3B8", linewidth=1.0, linestyle="--")
# Rolling mean of residuals to surface any trend in errors
resid_series = pd.Series(
residuals_df["residual"].values,
index=pd.to_datetime(residuals_df["date"])
).sort_index()
rolling_resid = resid_series.rolling(14, center=True).mean()
ax.plot(rolling_resid.index, rolling_resid.values,
color="#F59E0B", linewidth=2.0, label="14-day rolling mean of residuals")
ax.set_title("Residuals Over Time — All Categories (Test Period)", fontsize=12)
ax.set_xlabel("Date")
ax.set_ylabel("Residual (Actual − Predicted)")
ax.legend()
ax.grid(axis="y", linestyle="--", alpha=0.4)
plt.tight_layout()
plt.show()
print(f"Mean residual: {residuals_df['residual'].mean():.1f} (near zero = no systematic bias)")
print(f"Std of residuals: {residuals_df['residual'].std():.1f}")
A mean residual near zero indicates no systematic over- or under-prediction across the whole test period. The rolling mean chart should stay close to the zero line throughout July–October, then widen slightly in November–December as the Q4 spike introduces harder-to-predict days.
Business Interpretation¶
Three plain-English takeaways for the inventory or operations team:
1. Forecast accuracy is strong enough for weekly inventory ordering. At a MAPE of ~5.7%, the model is off by roughly 1 in 20 units on average. For inventory decisions made one week in advance, this provides meaningful signal — better than human judgment alone and much better than using the prior year's actuals without adjustment.
2. Forecast accuracy degrades in Q4. November and December errors are 60–70% higher than the mid-year baseline. The model cannot anticipate promotion-driven spikes because no promotion data was provided. The operations team should treat Q4 forecasts as a floor and apply a manual uplift based on the planned promotion calendar.
3. The lag-7 feature is the engine. Knowing what happened one week ago is the single most predictive input. This means forecast quality degrades for products that experience sudden structural breaks — a stockout, a viral social media moment, a competitor exit — because the "same day last week" anchor becomes misleading. Monitor lag-7 values for anomalies before trusting the forecast.
Limitations¶
- No holiday flags. Thanksgiving, Christmas Eve, and New Year's Day create spikes the model treats as ordinary late-year days. Adding a binary
is_holidaycolumn would directly address the Q4 error concentration. - No promotion data. Promotions cause sharp, temporary demand spikes that look like noise to this model but are entirely predictable with a promotion calendar.
- Cannot extrapolate. Tree-based models cap predictions at the maximum value seen during training. Multi-year forecasts will be progressively underestimated as the business grows.
- Category-level only. This forecast is at the category-day level. SKU-level forecasting (individual products) is substantially harder — many SKUs have intermittent demand with many zero-sales days, which MAPE handles poorly.
- No external signals. Weather, competitor pricing, consumer confidence, and search trends all influence sales. None are available in this dataset.
What Would You Do Next¶
- Add a holiday calendar — a
is_holidaybinary flag for major retail holidays (Thanksgiving, Christmas, Black Friday) is cheap to create and directly targets the Q4 error. - Add promotion flags — if the business has a promotion calendar, a
is_promotional_dayfeature would reduce the remaining hard-to-predict spikes. - Add exogenous regressors — weather data (for Food and Home categories) or consumer confidence indices are publicly available and have documented lift in retail forecasting.
- Try hierarchical forecasting — fit one model per category rather than a single model for all four. Category-specific models can learn category-specific seasonality patterns that a single model averages away.
- Extend the forecast horizon — this model was evaluated one step ahead (test period immediately follows training period). Evaluate multi-step performance (forecasting 30 or 90 days out) to understand how accuracy degrades with horizon length.