Day 03 — Part 2: Model Evaluation¶
A model that achieves 99% accuracy on a fraud dataset and catches 0% of actual fraud has failed completely. Evaluation is not a formality you run after training — it is the mechanism that tells you whether your model does what the business actually needs. This session covers how to measure performance honestly, choose the right metric, and tune models without fooling yourself.
Session Overview
Difficulty: Intermediate Reading time: ~2.5 hours | Exercises: ~2 hours Prerequisites: Feature Engineering (Day 03 Part 1) · Classification metrics (precision, recall, F1) · Regression metrics (MAE, RMSE, R²) · Scikit-learn workflow
What You Will Learn¶
- Why evaluation is harder than it looks, and where it goes wrong
- How cross-validation gives you a reliable estimate of generalisation performance
- Which regression metrics to use and when
- How to read a confusion matrix, ROC curve, and precision-recall curve
- How to tune hyperparameters without leaking information from the test set
Session Map¶
| # | Topic | File |
|---|---|---|
| 1 | Evaluation Overview | 01-evaluation-overview.md |
| 2 | Cross-Validation | 02-cross-validation.md |
| 3 | Regression Evaluation | 03-regression-evaluation.md |
| 4 | Classification Evaluation | 04-classification-evaluation.md |
| 5 | Model Selection and Tuning | 05-model-selection-and-tuning.md |
| 6 | Exercises | 06-exercises.md |
Estimated Time¶
| Activity | Time |
|---|---|
| Reading all notes | 60–75 min |
| Code walkthroughs | 30 min |
| Exercises | 60–90 min |
Difficulty: Intermediate. You need familiarity with scikit-learn, train/test splits, and at least one classification or regression model.
Prerequisites¶
Before starting this session, you should be comfortable with:
- Fitting models with scikit-learn (
fit,predict) - Train/test splits (
train_test_split) - Basic pandas for data manipulation
- Week 01 statistics — mean, variance, distributions
Start here: 01-evaluation-overview