Classification Evaluation

A model that predicts "no cancer" for every patient achieves 99% accuracy on a dataset where 1% of patients have cancer. That model is not useful — it is dangerous. Classification evaluation is about understanding which types of errors your model makes, and whether those errors are acceptable given the cost of being wrong.

Learning Objectives

• Build and read a confusion matrix
• Calculate precision, recall, and F1 score and explain the business meaning of each
• Choose the right averaging strategy for multiclass problems
• Interpret a ROC curve and understand what AUC actually measures
• Explain why the PR curve is often more informative than ROC for imbalanced data
• Tune the classification threshold based on business requirements

---

The Confusion Matrix

The confusion matrix breaks down predictions into four categories. Everything else in classification evaluation derives from these four numbers.

                    Predicted Positive    Predicted Negative
Actual Positive         TP                     FN
Actual Negative         FP                     TN

• True Positive (TP): correctly predicted positive — caught a fraud case
• True Negative (TN): correctly predicted negative — correctly cleared a legitimate transaction
• False Positive (FP): predicted positive, actually negative — flagged a legitimate transaction as fraud (false alarm)
• False Negative (FN): predicted negative, actually positive — missed an actual fraud case (missed detection)

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import (
    confusion_matrix,
    classification_report,
    ConfusionMatrixDisplay
)
import matplotlib.pyplot as plt

# Simulate a fraud detection dataset: 5% fraud
np.random.seed(42)
X_fraud, y_fraud = make_classification(
    n_samples=5000,
    n_features=20,
    weights=[0.95, 0.05],   # 95% legitimate, 5% fraud
    flip_y=0.02,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X_fraud, y_fraud, test_size=0.2, random_state=42, stratify=y_fraud
)

fraud_detector = GradientBoostingClassifier(n_estimators=100, random_state=42)
fraud_detector.fit(X_train, y_train)
y_pred = fraud_detector.predict(X_test)

# Build the confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)
# Output:
# [[936  11]
#  [ 14  39]]
# 
# TN=936, FP=11 (false alarms)
# FN=14 (missed fraud), TP=39 (caught fraud)

# Visualise it
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["Legitimate", "Fraud"])
disp.plot(cmap="Blues")
plt.title("Fraud Detection — Confusion Matrix")
plt.tight_layout()
plt.savefig("confusion_matrix.png", dpi=150)
plt.show()

---

Precision, Recall, and F1

Precision

Precision asks: of all the cases the model flagged as positive, what fraction were actually positive?

Formula: Precision = TP / (TP + FP)

If precision is 0.78 for fraud detection: when the model says "fraud", it is right 78% of the time. The other 22% are false alarms — legitimate transactions that get blocked.

When precision matters: when false positives are expensive. Spam filtering: marking a legitimate email as spam is worse than letting spam through. Credit scoring: incorrectly denying credit to a good customer is costly and legally risky.

Recall (Sensitivity)

Recall asks: of all the actual positive cases, what fraction did the model catch?

Formula: Recall = TP / (TP + FN)

If recall is 0.74 for fraud detection: the model catches 74% of actual fraud. The other 26% goes undetected.

When recall matters: when false negatives are expensive. Cancer screening: missing a cancer is far worse than triggering additional tests. Fraud detection: every missed fraud case is a direct financial loss.

F1 Score

F1 is the harmonic mean of precision and recall. It balances both — a high F1 requires both precision and recall to be high.

Formula: F1 = 2 * (Precision * Recall) / (Precision + Recall)

from sklearn.metrics import precision_score, recall_score, f1_score

precision = precision_score(y_test, y_pred)
recall    = recall_score(y_test, y_pred)
f1        = f1_score(y_test, y_pred)

print(f"Precision: {precision:.3f}")
print(f"Recall:    {recall:.3f}")
print(f"F1 Score:  {f1:.3f}")
print(f"Accuracy:  {(y_test == y_pred).mean():.3f}")
# Output: Precision: 0.780
# Output: Recall:    0.736
# Output: F1 Score:  0.757
# Output: Accuracy:  0.975  ← high because the majority class dominates

Notice that accuracy is 97.5% and F1 for the fraud class is 0.757. These two numbers describe very different things. Report accuracy to a business stakeholder on this problem and you will mislead them.

Warning

The precision-recall tradeoff is real and unavoidable. Making a model more aggressive (lower threshold) increases recall but decreases precision. You cannot improve both simultaneously without better features or a better model. The business must decide which error is more expensive.

The Full Classification Report

print(classification_report(y_test, y_pred, target_names=["Legitimate", "Fraud"]))
# Output:
#               precision    recall  f1-score   support
#
#   Legitimate       0.985     0.988     0.987       947
#        Fraud       0.780     0.736     0.757        53
#
#     accuracy                           0.975      1000
#    macro avg       0.883     0.862     0.872      1000
# weighted avg       0.975     0.975     0.975      1000

Reading this report:

• Each row is one class
• support is the number of actual cases in the test set for that class
• macro avg — the unweighted average across classes (treats each class equally)
• weighted avg — average weighted by support (majority class dominates this number)
• For imbalanced problems, always read the minority class row directly — not the averages

Info

The weighted avg row in classification_report is dominated by the majority class. On a 95/5 split, a model that ignores the minority class entirely can show weighted avg F1 = 0.96. Always inspect each class individually when classes are imbalanced.

---

Averaging Strategies for Multiclass Problems

When there are more than two classes, precision, recall, and F1 need to be collapsed across all classes.

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import f1_score

iris = load_iris()
X_iris, y_iris = iris.data, iris.target  # 3 classes: setosa, versicolor, virginica

X_train_iris, X_test_iris, y_train_iris, y_test_iris = train_test_split(
    X_iris, y_iris, test_size=0.2, random_state=42, stratify=y_iris
)

iris_model = RandomForestClassifier(n_estimators=50, random_state=42)
iris_model.fit(X_train_iris, y_train_iris)
y_pred_iris = iris_model.predict(X_test_iris)

# Macro: each class weighted equally
f1_macro = f1_score(y_test_iris, y_pred_iris, average="macro")

# Weighted: each class weighted by its support (number of samples)
f1_weighted = f1_score(y_test_iris, y_pred_iris, average="weighted")

# Micro: compute TP, FP, FN globally across all classes
f1_micro = f1_score(y_test_iris, y_pred_iris, average="micro")

print(f"F1 Macro:    {f1_macro:.3f}")
print(f"F1 Weighted: {f1_weighted:.3f}")
print(f"F1 Micro:    {f1_micro:.3f}")
# Output: F1 Macro:    0.967
# Output: F1 weighted: 0.967
# Output: F1 Micro:    0.967   (they converge when classes are balanced)

Which to use:

Dataset	Averaging strategy
Balanced classes	All three give similar results; weighted is conventional
Imbalanced classes, minority class matters	macro — treats each class equally
Imbalanced classes, performance on common cases matters	weighted
Binary classification	Do not use averaging — report for the positive class directly

---

ROC Curve and AUC

The ROC (Receiver Operating Characteristic) curve shows the tradeoff between the True Positive Rate (recall) and the False Positive Rate across all possible classification thresholds.

True Positive Rate (TPR): TP / (TP + FN) — same as recall
False Positive Rate (FPR): FP / (FP + TN) — rate of false alarms among actual negatives

As you lower the threshold, you catch more positives (higher TPR) but also generate more false alarms (higher FPR). The ROC curve traces this tradeoff.

from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt

# Get probability scores instead of binary predictions
y_proba = fraud_detector.predict_proba(X_test)[:, 1]  # probability of fraud

fpr, tpr, thresholds = roc_curve(y_test, y_proba)
auc_score = roc_auc_score(y_test, y_proba)

plt.figure(figsize=(7, 5))
plt.plot(fpr, tpr, color="steelblue", lw=2,
         label=f"Fraud Detector (AUC = {auc_score:.3f})")
plt.plot([0, 1], [0, 1], color="gray", linestyle="--", lw=1, label="Random classifier")
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate (Recall)")
plt.title("ROC Curve — Fraud Detection")
plt.legend()
plt.tight_layout()
plt.savefig("roc_curve.png", dpi=150)
plt.show()

print(f"AUC: {auc_score:.3f}")
# Output: AUC: 0.951

Interpreting AUC:

• AUC = 1.0 — perfect classifier
• AUC = 0.5 — no better than random guessing (the diagonal line)
• AUC = 0.9+ — excellent discrimination
• AUC = 0.7–0.9 — good discrimination
• AUC < 0.7 — poor discrimination, reconsider features or model

AUC measures the probability that the model ranks a randomly chosen positive example higher than a randomly chosen negative example.

Tip

AUC is threshold-independent, which makes it useful for comparing models without committing to a threshold. However, once you deploy, you need a specific threshold. Use AUC for model selection, then separately optimise the threshold for deployment.

---

Precision-Recall Curve and PR-AUC

The ROC curve can be misleading on heavily imbalanced datasets. When the negative class dominates, the FPR denominator is very large, which makes the FPR look deceptively small even when there are many false alarms.

The Precision-Recall curve plots precision vs recall across thresholds — both metrics focus on the positive class, making it more sensitive to performance on the minority class.

from sklearn.metrics import precision_recall_curve, average_precision_score

precision_curve, recall_curve, pr_thresholds = precision_recall_curve(y_test, y_proba)
pr_auc = average_precision_score(y_test, y_proba)

# Baseline for PR curve: fraction of positives in the test set
baseline_pr = y_test.mean()

plt.figure(figsize=(7, 5))
plt.plot(recall_curve, precision_curve, color="darkorange", lw=2,
         label=f"Fraud Detector (PR-AUC = {pr_auc:.3f})")
plt.axhline(y=baseline_pr, color="gray", linestyle="--", lw=1,
            label=f"Baseline (positive rate = {baseline_pr:.2f})")
plt.xlabel("Recall")
plt.ylabel("Precision")
plt.title("Precision-Recall Curve — Fraud Detection")
plt.legend()
plt.tight_layout()
plt.savefig("pr_curve.png", dpi=150)
plt.show()

print(f"PR-AUC: {pr_auc:.3f}")
# Output: PR-AUC: 0.782

Info

The baseline for the PR curve is the positive class prevalence. A random classifier has PR-AUC equal to the fraction of positives in the dataset. On a 5% fraud dataset, a random classifier gets PR-AUC ≈ 0.05. A PR-AUC of 0.78 represents a massive improvement over that baseline.

ROC vs PR — when to use each:

Situation	Use
Balanced classes	ROC-AUC is reliable
Imbalanced classes, minority class matters	PR-AUC — it amplifies differences in minority-class performance
Need a single number for model comparison	Both; report both if possible
Communicating to a non-technical audience	Precision and recall directly

---

Threshold Selection

The default threshold for model.predict() in scikit-learn is 0.5. That threshold is almost never the right choice. The right threshold depends on the relative cost of false positives and false negatives.

import pandas as pd

# Evaluate precision and recall at different thresholds
threshold_analysis = []
for threshold in np.arange(0.1, 0.9, 0.05):
    y_pred_threshold = (y_proba >= threshold).astype(int)
    prec = precision_score(y_test, y_pred_threshold, zero_division=0)
    rec  = recall_score(y_test, y_pred_threshold)
    f1   = f1_score(y_test, y_pred_threshold, zero_division=0)
    threshold_analysis.append({
        "threshold": round(threshold, 2),
        "precision": round(prec, 3),
        "recall":    round(rec, 3),
        "f1":        round(f1, 3)
    })

df_thresholds = pd.DataFrame(threshold_analysis)
print(df_thresholds.to_string(index=False))
# Output:
#  threshold  precision  recall     f1
#       0.10      0.056   1.000  0.106   ← catches everything, massive false alarm rate
#       0.20      0.115   0.962  0.205
#       0.30      0.286   0.906  0.434
#       0.40      0.494   0.849  0.625
#       0.50      0.780   0.736  0.757   ← scikit-learn default
#       0.60      0.867   0.604  0.712
#       0.70      0.935   0.509  0.659
#       0.80      1.000   0.321  0.486   ← only flags sure bets, misses most fraud

How to pick the threshold:

• If false negatives are expensive (missed cancer, missed fraud): lower the threshold, accepting more false alarms
• If false positives are expensive (false drug test, wrongful denial): raise the threshold
• If both matter equally: pick the threshold that maximises F1

# Find the threshold that maximises F1
best_row = df_thresholds.loc[df_thresholds["f1"].idxmax()]
print(f"Best F1 threshold: {best_row['threshold']}")
print(f"At this threshold — Precision: {best_row['precision']}, Recall: {best_row['recall']}, F1: {best_row['f1']}")
# Output: Best F1 threshold: 0.5
# Output: At this threshold — Precision: 0.780, Recall: 0.736, F1: 0.757

Warning

Threshold optimisation must happen on the validation set, not the test set. If you tune the threshold on the test set and then report test set performance at that threshold, you have overfit your reporting. Use cross-validation or a dedicated validation set to select the threshold, then apply it once on the test set.

Success

The key decision in classification evaluation is: which error is more expensive? Define that before you look at any metrics. If you define it after, you will unconsciously tune everything toward a number that looks good rather than a model that works.

---

---

What's Next

You've covered confusion matrix anatomy, precision and recall as complementary tradeoffs, F1-score and weighted averaging for multiclass problems, ROC-AUC for threshold-independent model comparison, PR-AUC for imbalanced class settings, and threshold selection based on the relative cost of false positives versus false negatives. Next up: 05-model-selection-and-tuning — where you'll use GridSearchCV and RandomizedSearchCV to systematically find the best hyperparameter combinations, compare multiple algorithms on a shared evaluation protocol, and make the final deployment decision based on cross-validated performance rather than test set experimentation.

Optional Deep Dive

Read the Fawcett (2006) paper "An Introduction to ROC Analysis" (Pattern Recognition Letters, free PDF widely available) — it provides the formal derivation of the ROC curve, the AUC interpretation as a ranking probability, and the class-imbalance argument for why PR-AUC is often more informative than ROC-AUC when the positive class is rare.

Previous: Regression Evaluation | Next: Model Selection and Tuning