Evaluation Overview¶
Your model's training accuracy is almost meaningless. What matters is how it performs on data it has never seen, measured by a metric that actually reflects what the business cares about. Getting both of those wrong is more common than getting them right — and the consequences show up in production, not in your notebook.
Learning Objectives¶
- Understand the three core questions that evaluation is trying to answer
- Recognise the failure modes that make evaluation dishonest
- Set up a baseline model before training anything complex
- Interpret evaluation results in terms the business can act on
The Three Questions Evaluation Answers¶
Every evaluation exercise is trying to answer three questions. If you cannot answer all three, your evaluation is incomplete.
1. Does the model generalise? A model that memorises training data is useless. Generalisation is the gap between training performance and test performance. If training accuracy is 98% and test accuracy is 71%, the model has overfit and will fail in production.
2. Is it better than the baseline? The baseline is the dumbest useful predictor — predict the mean for regression, predict the majority class for classification. If your model cannot beat the baseline, it has learned nothing that the distribution itself does not already tell you.
3. Is it good enough for the task? "Better than baseline" is not the same as "good enough to deploy". An F1 score of 0.62 might be transformative in one domain and completely insufficient in another. The threshold for "good enough" comes from the business, not from the model.
Why Evaluation Goes Wrong¶
Overfitting to the Test Set¶
The test set is supposed to simulate future, unseen data. The moment you use the test set to make a decision — adjust a threshold, pick a model, tune a feature — it stops being a fair estimate of future performance. Every look at the test set uses up some of its information.
Warning
Never tune anything based on test set performance. Use a validation set or cross-validation for all decisions. The test set is touched exactly once: to report the final result.
Distribution Shift¶
Your test set reflects the world as it was when you collected the data. If the world changes — seasonality, economic shifts, new user behaviour — the model's real-world performance will diverge from what the test set showed. This is called distribution shift, and no metric can detect it automatically.
Info
Distribution shift is why models degrade over time in production. A model trained on pre-pandemic purchasing behaviour will behave differently post-pandemic, even if it evaluated perfectly on a held-out set from the same period.
Metric-Objective Mismatch¶
The metric you optimise during training is not always the metric the business actually cares about. Optimising accuracy on an imbalanced fraud dataset rewards the model for predicting "not fraud" every time. Optimising RMSE for a sales forecast punishes large errors heavily — which may or may not match what the business actually loses when a large order goes wrong.
Warning
Choosing the wrong metric is as dangerous as building the wrong model. A model optimised for the wrong objective can be confidently, consistently wrong in exactly the situations that matter most.
Leakage¶
Data leakage is when information from the future or from the target variable sneaks into your features. The model learns to exploit it during training, scores brilliantly on your test set, and fails immediately in production where that information does not exist.
Common leakage patterns: - A feature derived from the target (e.g., a running total that includes the current row's value) - Future data in a temporal dataset (e.g., next week's price in a model that predicts next week's price) - Normalisation or scaling fitted on the full dataset before splitting
Always Start with a Baseline¶
The baseline is your reality check. It answers the question: how well can we do without any machine learning at all?
Regression baseline — predict the mean of the training target for every row:
import numpy as np
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_absolute_error
# Toy example: predicting house prices
y_train = np.array([200000, 250000, 180000, 310000, 225000])
y_test = np.array([215000, 260000, 195000])
baseline = DummyRegressor(strategy="mean")
baseline.fit(y_train.reshape(-1, 1), y_train)
y_baseline_pred = baseline.predict(np.zeros((len(y_test), 1)))
print(f"Baseline MAE: {mean_absolute_error(y_test, y_baseline_pred):,.0f}")
# Output: Baseline MAE: 26,667
Classification baseline — predict the majority class for every row:
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, f1_score
# Fraud detection: 97% legitimate, 3% fraud
y_train_labels = [0] * 970 + [1] * 30 # 1000 samples
y_test_labels = [0] * 97 + [1] * 3 # 100 samples
baseline_clf = DummyClassifier(strategy="most_frequent")
baseline_clf.fit([[0]] * len(y_train_labels), y_train_labels)
y_baseline = baseline_clf.predict([[0]] * len(y_test_labels))
print(f"Baseline Accuracy: {accuracy_score(y_test_labels, y_baseline):.2%}")
print(f"Baseline F1 (fraud class): {f1_score(y_test_labels, y_baseline, pos_label=1):.2f}")
# Output: Baseline Accuracy: 97.00%
# Output: Baseline F1 (fraud class): 0.00
This is the trap. The majority-class classifier achieves 97% accuracy and catches zero fraud. If you report accuracy, the baseline looks excellent. That is why metric choice matters.
Success
A model that cannot beat a majority-class classifier or a mean predictor has not learned anything from your features. The baseline is not a low bar — it is the minimum bar.
What Good Evaluation Looks Like in Practice¶
A complete evaluation report for a model answers:
| Question | What to report |
|---|---|
| What problem does the model solve? | Task description, business context |
| What data was used? | Size, time period, class balance |
| What baseline was used? | Dummy strategy and score |
| What metric was used and why? | Metric name, rationale tied to business cost |
| What is the cross-validation score? | Mean ± std across folds |
| What is the test set score? | Single number, reported once |
| Where does the model fail? | Error analysis — which segments, which cases |
| Is it good enough to deploy? | A yes/no based on a pre-agreed threshold |
Tip
Decide what "good enough" means before you train anything. If you define the threshold after seeing the results, you will unconsciously move the goalposts.
Error Analysis¶
Aggregate metrics hide a lot. A model with a 0.80 F1 score might be excellent for the majority of cases and completely wrong for the cases that matter most.
import pandas as pd
from sklearn.metrics import mean_absolute_error
# After predictions are made
results = pd.DataFrame({
"actual": y_test,
"predicted": y_pred,
"error": y_test - y_pred,
"abs_error": abs(y_test - y_pred),
"segment": segment_labels # e.g., "high_value", "low_value"
})
# Where are the biggest errors?
print(results.nlargest(10, "abs_error")[["actual", "predicted", "error", "segment"]])
# Is error distributed evenly across segments?
print(results.groupby("segment")["abs_error"].mean())
Look for patterns: does the model systematically underpredict for one customer segment? Overpredict at high values? These patterns are actionable — aggregate MAE is not.
Info
Error analysis is where you discover that your model works for 85% of cases and is badly wrong for the other 15% — which might be the 15% that drives 60% of revenue.
What's Next¶
You've covered why evaluation matters, the train/validation/test protocol, baseline model comparison, the cost-structure framework for choosing metrics, systematic error analysis by segment, and the diagnostic pattern of slicing predictions to find where the model systematically fails. Next up: 02-cross-validation — where you'll implement stratified K-Fold, time series splits, and nested cross-validation, and learn why the mean ± std pattern is the standard way to report model performance.
Optional Deep Dive
Read "Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning" by Sebastian Raschka (free at arxiv.org/abs/1811.12808) — it covers cross-validation strategies, the statistical tests for comparing models, and nested CV in rigorous mathematical detail, giving you the theoretical foundation behind the practical techniques covered here.