Classification Metrics — Measuring What Actually Matters¶
Accuracy is the most misleading metric in machine learning. A fraud detection model that flags nothing and calls every transaction legitimate will achieve 99.9% accuracy on most datasets — because fraud is rare. That model is catastrophically wrong. Choosing the right metric is not a technical afterthought; it is a business decision that determines whether your model is actually useful.
Learning Objectives¶
- Explain why accuracy fails on imbalanced classification problems
- Read a confusion matrix and identify TP, FP, TN, FN
- Compute and interpret precision, recall, F1-score, and ROC-AUC
- Know when to use PR-AUC instead of ROC-AUC
- Adjust the decision threshold to optimise for a specific business objective
- Explain the precision-recall tradeoff to a non-technical stakeholder
Why Accuracy Is Not Enough¶
Consider a credit card fraud detection system. In a typical dataset, 0.1% of transactions are fraudulent. A model that predicts "legitimate" for every single transaction achieves 99.9% accuracy. It is also completely useless — it catches zero fraud.
import numpy as np
from sklearn.metrics import accuracy_score, classification_report
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import train_test_split
# Simulated fraud dataset: 0.5% fraud rate
np.random.seed(42)
n_transactions = 10000
fraud_rate = 0.005
y_fraud = np.random.choice(
[0, 1], size=n_transactions, p=[1 - fraud_rate, fraud_rate]
)
X_fraud = np.random.randn(n_transactions, 10)
X_tr, X_te, y_tr, y_te = train_test_split(
X_fraud, y_fraud, test_size=0.2, random_state=42
)
# Dummy model that always predicts "legitimate"
always_legit = DummyClassifier(strategy="most_frequent")
always_legit.fit(X_tr, y_tr)
dummy_preds = always_legit.predict(X_te)
print(f"Dummy classifier accuracy: {accuracy_score(y_te, dummy_preds):.4f}")
print(f"Fraud cases in test set: {y_te.sum()}")
print(f"Fraud cases caught: {(dummy_preds[y_te == 1] == 1).sum()}")
# Output:
# Dummy classifier accuracy: 0.9940
# Fraud cases in test set: 9
# Fraud cases caught: 0
Warning
When classes are imbalanced, accuracy tells you how often the model predicts the majority class correctly — which is trivial. Switch to precision, recall, F1, or AUC as your primary metrics for any classification problem with imbalanced classes.
The Confusion Matrix — The Foundation of Everything¶
Every classification metric derives from the confusion matrix. For binary classification:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actually Positive | True Positive (TP) | False Negative (FN) |
| Actually Negative | False Positive (FP) | True Negative (TN) |
- TP: Model said positive, it is positive. Correct catch.
- TN: Model said negative, it is negative. Correct clear.
- FP: Model said positive, it is negative. False alarm.
- FN: Model said negative, it is positive. Missed case.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
import matplotlib.pyplot as plt
cancer = load_breast_cancer(as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
cancer.data, cancer.target,
test_size=0.2, random_state=42, stratify=cancer.target
)
rf_clf = Pipeline([
("scaler", StandardScaler()),
("model", RandomForestClassifier(n_estimators=200, random_state=42))
])
rf_clf.fit(X_train, y_train)
y_pred = rf_clf.predict(X_test)
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)
# Output:
# Confusion Matrix:
# [[40 3]
# [ 0 71]]
# Rows are actual: [malignant, benign]
# Columns are predicted: [malignant, benign]
# TP (malignant caught) = 40, FN (malignant missed) = 3
# FP (benign called malignant) = 0, TN (benign correctly cleared) = 71
# Visual version
disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=["malignant", "benign"])
disp.plot(cmap="Blues")
plt.title("Random Forest — Confusion Matrix")
plt.tight_layout()
plt.savefig("confusion_matrix.png", dpi=150)
Info
Reading a confusion matrix: rows represent actual class, columns represent predicted class. The diagonal is always correct predictions. Off-diagonal elements are errors. A cell at row i, column j means "the model predicted class j for a sample that was actually class i."
Precision — The Cost of False Alarms¶
Precision answers: Of all the cases the model predicted as positive, what fraction were actually positive?
$$\text{Precision} = \frac{TP}{TP + FP}$$
High precision matters when false positives are expensive. In spam detection, marking a legitimate email as spam (FP) is disruptive. You want every email in the spam folder to actually be spam.
from sklearn.metrics import precision_score
precision = precision_score(y_test, y_pred)
print(f"Precision: {precision:.4f}")
# Output: Precision: 1.0000
# Every case the model called malignant was actually malignant (no false alarms)
Recall — The Cost of Missing Cases¶
Recall (also called Sensitivity or True Positive Rate) answers: Of all the actual positive cases, what fraction did the model correctly catch?
$$\text{Recall} = \frac{TP}{TP + FN}$$
High recall matters when false negatives are expensive. In cancer screening, missing a malignant tumour (FN) means a patient goes untreated. You want to catch every positive case, even at the cost of some false alarms.
from sklearn.metrics import recall_score
recall = recall_score(y_test, y_pred)
print(f"Recall: {recall:.4f}")
# Output: Recall: 0.9302
# The model caught 93% of malignant cases, missing 7%
The Precision-Recall Tradeoff¶
Precision and recall pull in opposite directions. As you lower the classification threshold, you predict more positives — catching more true positives (recall rises) but also flagging more false positives (precision falls).
import pandas as pd
from sklearn.metrics import precision_score, recall_score, f1_score
y_proba = rf_clf.predict_proba(X_test)[:, 1] # probability of being benign (class 1)
threshold_results = []
for threshold in np.arange(0.1, 1.0, 0.1):
y_pred_thresh = (y_proba >= threshold).astype(int)
threshold_results.append({
"Threshold": round(threshold, 1),
"Predicted Positive": int(y_pred_thresh.sum()),
"Precision": round(precision_score(y_test, y_pred_thresh, zero_division=0), 3),
"Recall": round(recall_score(y_test, y_pred_thresh, zero_division=0), 3),
"F1": round(f1_score(y_test, y_pred_thresh, zero_division=0), 3)
})
print(pd.DataFrame(threshold_results).to_string(index=False))
# Output (approximate):
# Threshold Predicted Positive Precision Recall F1
# 0.1 113 0.637 1.000 0.778
# 0.2 113 0.637 1.000 0.778
# 0.3 107 0.654 0.986 0.787
# 0.4 101 0.683 0.972 0.803
# 0.5 74 0.959 1.000 0.979
# 0.6 72 0.986 1.000 0.993
# 0.7 70 1.000 0.986 0.993
# 0.8 62 1.000 0.873 0.932
# 0.9 41 1.000 0.577 0.732
Tip
The precision_recall_curve function from sklearn gives you this tradeoff at every possible threshold. Plot it and find the threshold where precision and recall both meet your business requirements — not where F1 is maximised, but where the business problem is solved.
F1-Score — Balancing Precision and Recall¶
F1-score is the harmonic mean of precision and recall:
$$F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$
The harmonic mean is used instead of the arithmetic mean because it penalises imbalance between precision and recall. A model with precision=1.0 and recall=0.01 gets an arithmetic mean of 0.505 but an F1 of 0.02 — which correctly reflects that the model is nearly useless.
from sklearn.metrics import f1_score
f1 = f1_score(y_test, y_pred)
print(f"F1-score: {f1:.4f}")
# Output: F1-score: 0.9793
# Why harmonic mean: punishes extreme imbalance
prec, rec = 1.0, 0.01
arithmetic_mean = (prec + rec) / 2
harmonic_mean = 2 * (prec * rec) / (prec + rec)
print(f"Arithmetic mean: {arithmetic_mean:.4f}") # Output: 0.5050 — misleadingly high
print(f"Harmonic mean: {harmonic_mean:.4f}") # Output: 0.0198 — correctly reflects uselessness
Info
There is also F_beta score, which lets you weight recall more than precision (or vice versa). F2 weights recall twice as heavily as precision — useful when missing positives is more costly than false alarms. F0.5 does the reverse.
ROC-AUC — Threshold-Independent Ranking¶
The ROC curve (Receiver Operating Characteristic) plots the True Positive Rate (recall) against the False Positive Rate at every possible threshold. The AUC (Area Under the Curve) summarises the entire curve as a single number.
AUC = 0.5 means the model is no better than random guessing (the diagonal line). AUC = 1.0 means perfect separation — the model ranks every positive above every negative.
The key insight: AUC measures how well the model ranks predictions, not whether any specific threshold is good. A model with AUC=0.97 will rank a randomly chosen positive case above a randomly chosen negative case 97% of the time.
from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt
y_proba_positive = rf_clf.predict_proba(X_test)[:, 1]
auc_score = roc_auc_score(y_test, y_proba_positive)
print(f"ROC-AUC: {auc_score:.4f}")
# Output: ROC-AUC: 0.9979
fpr, tpr, roc_thresholds = roc_curve(y_test, y_proba_positive)
plt.figure(figsize=(7, 6))
plt.plot(fpr, tpr, color="teal", linewidth=2, label=f"ROC Curve (AUC = {auc_score:.3f})")
plt.plot([0, 1], [0, 1], "k--", alpha=0.4, label="Random baseline")
plt.xlabel("False Positive Rate (1 - Specificity)")
plt.ylabel("True Positive Rate (Recall)")
plt.title("ROC Curve — Random Forest")
plt.legend(loc="lower right")
plt.tight_layout()
plt.savefig("roc_curve.png", dpi=150)
PR-AUC — Better for Imbalanced Data¶
The Precision-Recall curve plots precision against recall at every threshold. The area under this curve (PR-AUC, also called Average Precision) is more informative than ROC-AUC when classes are highly imbalanced.
Why: ROC-AUC can look deceptively good even when the model is poor at finding the minority class, because the large TN count makes FPR look small. PR-AUC focuses only on the positive class — it cannot be inflated by a large number of true negatives.
from sklearn.metrics import average_precision_score, precision_recall_curve
ap_score = average_precision_score(y_test, y_proba_positive)
print(f"PR-AUC (Average Precision): {ap_score:.4f}")
# Output: PR-AUC (Average Precision): 0.9985
precision_curve, recall_curve, pr_thresholds = precision_recall_curve(y_test, y_proba_positive)
plt.figure(figsize=(7, 6))
plt.plot(recall_curve, precision_curve, color="teal", linewidth=2,
label=f"PR Curve (AP = {ap_score:.3f})")
plt.axhline(y_test.mean(), color="gray", linestyle="--", alpha=0.7,
label=f"Baseline (prevalence = {y_test.mean():.2f})")
plt.xlabel("Recall")
plt.ylabel("Precision")
plt.title("Precision-Recall Curve — Random Forest")
plt.legend(loc="lower left")
plt.tight_layout()
plt.savefig("pr_curve.png", dpi=150)
Warning
The baseline for PR-AUC is the class prevalence (fraction of positives in the dataset), not 0.5. A model with PR-AUC equal to the prevalence is no better than random. A model detecting 1% fraud that achieves PR-AUC of 0.70 is genuinely impressive — it is predicting rare events with high reliability.
The Full Classification Report¶
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred, target_names=["malignant", "benign"]))
# Output:
# precision recall f1-score support
#
# malignant 1.00 0.93 0.96 43
# benign 0.96 1.00 0.98 71
#
# accuracy 0.97 114
# macro avg 0.98 0.97 0.97 114
# weighted avg 0.98 0.97 0.97 114
How to read this: - malignant row: Of predicted malignant cases, 100% were truly malignant (precision=1.00). Of actual malignant cases, 93% were caught (recall=0.93). - macro avg: Unweighted average across classes — treats each class equally regardless of size. - weighted avg: Weighted by class support — gives more weight to the larger class.
Success
Always print the full classification report, not just accuracy. The report forces you to look at per-class performance. A single number like "97% accuracy" hides the fact that you might be perfect on one class and mediocre on another.
Choosing the Right Metric¶
| Business Problem | Primary Metric | Reason |
|---|---|---|
| Fraud detection | Recall + Precision / PR-AUC | Missing fraud (FN) is costly; FP triggers investigations |
| Cancer screening | Recall (maximise) | Missing a case (FN) is catastrophic |
| Spam filter | Precision (maximise) | Sending real email to spam (FP) destroys trust |
| Churn prediction | F1 or PR-AUC | Both precision and recall matter; classes often imbalanced |
| Credit risk scoring | ROC-AUC | Ranking quality across thresholds matters |
| Balanced classification | Accuracy or F1 | When both classes matter equally |
Threshold Optimisation for a Business Goal¶
The right threshold is determined by the business, not the algorithm. Here is a systematic approach:
from sklearn.metrics import precision_recall_curve
precision_vals, recall_vals, thresh_vals = precision_recall_curve(y_test, y_proba_positive)
# Scenario: We are doing cancer screening.
# Business requirement: catch at least 98% of malignant cases (recall >= 0.98)
# Subject to that constraint, maximise precision.
min_required_recall = 0.98
eligible_mask = recall_vals[:-1] >= min_required_recall # exclude last point (no threshold)
if eligible_mask.any():
best_idx = np.where(eligible_mask)[0][np.argmax(precision_vals[:-1][eligible_mask])]
best_threshold = thresh_vals[best_idx]
best_precision = precision_vals[best_idx]
best_recall = recall_vals[best_idx]
print(f"Optimal threshold for recall >= {min_required_recall}:")
print(f" Threshold: {best_threshold:.3f}")
print(f" Precision: {best_precision:.4f}")
print(f" Recall: {best_recall:.4f}")
# Output (approximate):
# Optimal threshold for recall >= 0.98:
# Threshold: 0.330
# Precision: 0.9836
# Recall: 0.9859
# Apply this threshold to get predictions
y_pred_optimised = (y_proba_positive >= best_threshold).astype(int)
print(f"\nWith optimised threshold:")
print(classification_report(y_test, y_pred_optimised, target_names=["malignant", "benign"]))
Interview Questions¶
Q: Why does accuracy fail as a metric for imbalanced classification?
Show answer
On an imbalanced dataset (e.g., 95% class 0, 5% class 1), a model that always predicts class 0 achieves 95% accuracy without learning anything. Accuracy rewards correctly predicting the majority class, which is trivial. It does not capture how well the model performs on the minority class, which is usually the class of interest. Use precision, recall, F1, or AUC instead.
Q: What is the difference between precision and recall?
Show answer
Precision measures quality of positive predictions: of all predictions made as positive, what fraction were correct? It answers "when the model says yes, how often is it right?" Recall measures coverage of actual positives: of all actual positive cases, what fraction did the model catch? It answers "of all the real positives, how many did we find?" Precision penalises false alarms; recall penalises missed cases.
Q: Why is F1 the harmonic mean and not the arithmetic mean?
Show answer
The harmonic mean penalises extreme imbalance between precision and recall. If a model has precision=1.0 and recall=0.01, the arithmetic mean is 0.505 — which sounds reasonable. The harmonic mean gives 0.0198 — which correctly reflects that the model is nearly useless because it misses 99% of positives. The harmonic mean is always ≤ arithmetic mean, and approaches zero when either precision or recall is near zero.
Q: When would you choose PR-AUC over ROC-AUC?
Show answer
When the positive class is rare (imbalanced dataset). ROC-AUC is computed from TPR and FPR. FPR uses TN in the denominator — when there are many negatives (as in fraud detection), even a model that flags many false positives will have a low FPR because TN is huge. This makes ROC-AUC look better than it is. PR-AUC focuses only on the positive class (precision and recall), so a large number of true negatives cannot inflate it. For any problem where you care primarily about detecting a rare class, PR-AUC is the more honest metric.
Q: A model achieves 0.96 ROC-AUC but your manager wants to know if it will work. What else do you check?
Show answer
ROC-AUC is a ranking metric — it tells you the model is good at ordering positives above negatives. But your manager probably cares about operational performance at a specific operating point: what precision and recall do we get at the threshold we actually deploy? Print the precision-recall curve, pick a threshold that meets the business requirements (e.g., recall ≥ 0.95 for cancer screening), and report precision, recall, F1, and expected number of cases caught and false alarms per day.