Skip to content

Exercises — Classification Algorithms

These exercises build on everything from this session. Each exercise has three levels: warm-up establishes the baseline, main challenges your understanding, and stretch pushes you into territory practitioners actually face on real projects.

Work through them in order. The churn dataset thread across Exercises 1–3 mirrors how a real classification project would unfold.

Exercise 1 — Warm-up: Your First Classifier Pipeline

Dataset: load_breast_cancer() from sklearn
Goal: Build a working Logistic Regression pipeline and read the output correctly.

Tasks:

  1. Load the dataset and inspect the class distribution with pd.Series(y).value_counts()
  2. Split into train and test with stratify=y and test_size=0.2
  3. Build a pipeline: StandardScaler → LogisticRegression(max_iter=1000)
  4. Train and print:
  5. Test accuracy
  6. Full classification_report
  7. First 5 predicted probabilities alongside the true labels

Expected output (approximate): 

Accuracy: 0.9737
              precision    recall  f1-score   support
   malignant       0.98      0.95      0.96        43
      benign       0.97      0.99      0.98        71
    accuracy                           0.97       114

Show solution 
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score, classification_report
import pandas as pd

cancer = load_breast_cancer(as_frame=True)
X, y = cancer.data, cancer.target

print("Class distribution:")
print(pd.Series(y).value_counts())
# 1 (benign)    357
# 0 (malignant) 212

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

log_reg_pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=1000))
])
log_reg_pipe.fit(X_train, y_train)

y_pred = log_reg_pipe.predict(X_test)
y_proba = log_reg_pipe.predict_proba(X_test)[:, 1]

print(f"\nAccuracy: {accuracy_score(y_test, y_pred):.4f}")
print(classification_report(y_test, y_pred, target_names=["malignant", "benign"]))

# First 5 predictions
comparison = pd.DataFrame({
    "True Label": y_test.values[:5],
    "Predicted": y_pred[:5],
    "P(benign)": y_proba[:5].round(3)
})
print(comparison)

Exercise 2 — Main: Compare Four Classifiers on Churn Data

Dataset: Simulated churn dataset (generate with the code below)
Goal: Train four classifiers, compare them with cross-validation, and identify which metric matters most for churn.

Tasks:

  1. Generate the churn dataset using the starter code
  2. Train with 5-fold cross-validation: Logistic Regression, Decision Tree, Random Forest, Gradient Boosting
  3. Report mean and std for both accuracy and ROC-AUC
  4. Identify the best model by AUC
  5. Answer in a comment: why is ROC-AUC more informative than accuracy here?

Starter code: 

from sklearn.datasets import make_classification

# Simulate a churn dataset: 10% churn rate, 15 features (some informative, some noise)
X_churn, y_churn = make_classification(
    n_samples=5000,
    n_features=15,
    n_informative=8,
    n_redundant=3,
    weights=[0.90, 0.10],    # 90% non-churn, 10% churn
    flip_y=0.05,             # 5% label noise (realistic)
    random_state=42
)

print(f"Class distribution: {dict(zip(*[range(2), [sum(y_churn==c) for c in range(2)]])}")
# Class distribution: {0: 4498, 1: 502}

Expected output (approximate): 

                 Model  Accuracy (mean)  Accuracy (std)  AUC (mean)  AUC (std)
  Logistic Regression           0.9020          0.0078      0.7891     0.0212
  Decision Tree                 0.8924          0.0102      0.6834     0.0289
  Random Forest                 0.9222          0.0066      0.8734     0.0143
  Gradient Boosting             0.9278          0.0058      0.8912     0.0121

Show solution 
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
import pandas as pd

X_churn, y_churn = make_classification(
    n_samples=5000,
    n_features=15,
    n_informative=8,
    n_redundant=3,
    weights=[0.90, 0.10],
    flip_y=0.05,
    random_state=42
)

classifiers = {
    "Logistic Regression": Pipeline([
        ("scaler", StandardScaler()),
        ("model", LogisticRegression(max_iter=1000, C=1.0))
    ]),
    "Decision Tree": DecisionTreeClassifier(max_depth=5, random_state=42),
    "Random Forest": RandomForestClassifier(n_estimators=200, n_jobs=-1, random_state=42),
    "Gradient Boosting": GradientBoostingClassifier(
        n_estimators=200, learning_rate=0.05, max_depth=3, random_state=42
    )
}

results = []
for name, clf in classifiers.items():
    acc_scores = cross_val_score(clf, X_churn, y_churn, cv=5, scoring="accuracy")
    auc_scores = cross_val_score(clf, X_churn, y_churn, cv=5, scoring="roc_auc")
    results.append({
        "Model": name,
        "Accuracy (mean)": round(acc_scores.mean(), 4),
        "Accuracy (std)": round(acc_scores.std(), 4),
        "AUC (mean)": round(auc_scores.mean(), 4),
        "AUC (std)": round(auc_scores.std(), 4)
    })

comparison_df = pd.DataFrame(results).sort_values("AUC (mean)", ascending=False)
print(comparison_df.to_string(index=False))

best_model_name = comparison_df.iloc[0]["Model"]
print(f"\nBest model by AUC: {best_model_name}")

# Why ROC-AUC is better here:
# The dataset has 90% non-churn. Accuracy rewards predicting 'no churn' for everyone,
# which achieves 90% accuracy trivially. ROC-AUC measures how well the model ranks
# churners above non-churners regardless of threshold — it cannot be gamed by
# predicting the majority class.

Exercise 3 — Main: Threshold Optimisation for Recall

Dataset: Continue from Exercise 2 (churn dataset)
Goal: Train the best-performing model from Exercise 2, then find the threshold that achieves at least 80% recall on churners while maximising precision.

Context: Your product team says: "We can only run retention campaigns for 200 customers per week. Flag the 200 most likely churners." This is a precision-optimisation problem at a fixed budget. Separately, the CFO says: "We must catch at least 80% of churners — any fewer and we lose too much revenue." This is a recall-constraint problem.

Tasks:

  1. Train the best model from Exercise 2 on an 80/20 train/test split
  2. Get predicted probabilities for the test set
  3. Find the threshold that maximises precision subject to recall ≥ 0.80
  4. Print the confusion matrix at this threshold
  5. Interpret: how many churners did you catch? How many non-churners did you incorrectly flag?
  6. Bonus: Plot the Precision-Recall curve and mark the chosen operating point

Expected output (approximate): 

Optimal threshold: 0.312
At this threshold:
  Precision: 0.508
  Recall:    0.800
  F1-score:  0.622

Confusion Matrix:
[[814  80]
 [ 20  86]]

Interpretation:
  Churners correctly caught (TP): 86 of 106
  Non-churners incorrectly flagged (FP): 80
  For every real churner found, we investigate 0.93 false alarms

Show solution 
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import (
    precision_recall_curve, confusion_matrix, ConfusionMatrixDisplay,
    precision_score, recall_score, f1_score
)
import numpy as np
import matplotlib.pyplot as plt

# Re-generate the dataset (or use from Exercise 2)
from sklearn.datasets import make_classification
X_churn, y_churn = make_classification(
    n_samples=5000, n_features=15, n_informative=8, n_redundant=3,
    weights=[0.90, 0.10], flip_y=0.05, random_state=42
)

X_tr, X_te, y_tr, y_te = train_test_split(
    X_churn, y_churn, test_size=0.2, random_state=42, stratify=y_churn
)

best_clf = GradientBoostingClassifier(
    n_estimators=200, learning_rate=0.05, max_depth=3, random_state=42
)
best_clf.fit(X_tr, y_tr)

churn_proba = best_clf.predict_proba(X_te)[:, 1]

# Find threshold: recall >= 0.80, maximise precision
precisions, recalls, thresholds = precision_recall_curve(y_te, churn_proba)

min_recall = 0.80
eligible = recalls[:-1] >= min_recall

if eligible.any():
    best_idx = np.where(eligible)[0][np.argmax(precisions[:-1][eligible])]
    optimal_threshold = thresholds[best_idx]
    optimal_precision = precisions[best_idx]
    optimal_recall = recalls[best_idx]

    print(f"Optimal threshold: {optimal_threshold:.3f}")
    print(f"At this threshold:")
    print(f"  Precision: {optimal_precision:.3f}")
    print(f"  Recall:    {optimal_recall:.3f}")

    y_pred_opt = (churn_proba >= optimal_threshold).astype(int)
    print(f"  F1-score:  {f1_score(y_te, y_pred_opt):.3f}")

    cm = confusion_matrix(y_te, y_pred_opt)
    print(f"\nConfusion Matrix:\n{cm}")

    TP = cm[1, 1]
    FP = cm[0, 1]
    FN = cm[1, 0]
    total_churners = cm[1].sum()
    print(f"\nInterpretation:")
    print(f"  Churners correctly caught (TP): {TP} of {total_churners}")
    print(f"  Non-churners incorrectly flagged (FP): {FP}")
    if TP > 0:
        print(f"  For every real churner found, we investigate {FP/TP:.2f} false alarms")

    # Plot PR curve with operating point
    plt.figure(figsize=(8, 6))
    plt.plot(recalls[:-1], precisions[:-1], color="teal", linewidth=2, label="PR Curve")
    plt.scatter([optimal_recall], [optimal_precision], s=100, color="red", zorder=5,
                label=f"Operating point (t={optimal_threshold:.2f})")
    plt.axhline(y_te.mean(), color="gray", linestyle="--", alpha=0.6,
                label=f"Baseline (prevalence={y_te.mean():.2f})")
    plt.xlabel("Recall")
    plt.ylabel("Precision")
    plt.title("Precision-Recall Curve with Chosen Operating Point")
    plt.legend()
    plt.tight_layout()
    plt.savefig("pr_curve_optimised.png", dpi=150)
    print("\nPR curve saved to pr_curve_optimised.png")

Exercise 4 — Stretch: Random Forest Feature Importance and Model Interpretation

Dataset: A simulated customer churn dataset with named features
Goal: Build an interpretable Random Forest, extract feature importances, and write a 3-sentence business interpretation of the most important drivers.

Tasks:

  1. Generate the dataset using the starter code (it includes named features)
  2. Train a Random Forest with n_estimators=300
  3. Plot a horizontal bar chart of the top 10 feature importances
  4. Compute permutation importances and compare them to the built-in importances
  5. Write a brief interpretation: which features drive churn in this model?

Starter code: 

import pandas as pd
import numpy as np
from sklearn.datasets import make_classification

np.random.seed(42)
n = 3000

# Simulated churn features
X_named = pd.DataFrame({
    "monthly_spend": np.random.gamma(3, 20, n),
    "contract_months": np.random.choice([1, 12, 24], n),
    "support_calls": np.random.poisson(2, n),
    "days_since_login": np.random.exponential(15, n),
    "num_products": np.random.randint(1, 6, n),
    "tenure_months": np.random.gamma(5, 10, n),
    "payment_failures": np.random.poisson(0.5, n),
    "satisfaction_score": np.random.randint(1, 6, n),
    "age": np.random.randint(18, 75, n),
    "region_encoded": np.random.randint(0, 5, n)
})

# Build churn label with known relationships
log_odds = (
    -3.0
    + 0.05 * X_named["support_calls"]
    + 0.04 * X_named["days_since_login"]
    + 1.50 * X_named["payment_failures"]
    - 0.30 * X_named["satisfaction_score"]
    - 0.02 * X_named["tenure_months"]
    + 0.01 * (X_named["contract_months"] == 1).astype(int)
)
churn_prob = 1 / (1 + np.exp(-log_odds))
y_named = (np.random.rand(n) < churn_prob).astype(int)

print(f"Churn rate: {y_named.mean():.1%}")
# Expected: Churn rate: ~15-20%

Expected outputs: 

Top 5 features by impurity importance:
    payment_failures: 0.2341
    satisfaction_score: 0.1987
    days_since_login: 0.1654
    tenure_months: 0.1123
    support_calls: 0.0987

Top 5 features by permutation importance:
    payment_failures: 0.0412 ± 0.0031
    satisfaction_score: 0.0387 ± 0.0028
    days_since_login: 0.0298 ± 0.0022
    ...

Business interpretation (sample): 

Payment failures are the strongest predictor of churn, suggesting that billing friction
drives customers out before they consciously decide to leave. Low satisfaction scores
compound this — customers who already have billing issues and are unsatisfied are at
very high churn risk. Retention campaigns should prioritise customers with any payment
failures in the last 90 days.

Show solution 
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance
from sklearn.metrics import classification_report, roc_auc_score

np.random.seed(42)
n = 3000

X_named = pd.DataFrame({
    "monthly_spend": np.random.gamma(3, 20, n),
    "contract_months": np.random.choice([1, 12, 24], n),
    "support_calls": np.random.poisson(2, n),
    "days_since_login": np.random.exponential(15, n),
    "num_products": np.random.randint(1, 6, n),
    "tenure_months": np.random.gamma(5, 10, n),
    "payment_failures": np.random.poisson(0.5, n),
    "satisfaction_score": np.random.randint(1, 6, n),
    "age": np.random.randint(18, 75, n),
    "region_encoded": np.random.randint(0, 5, n)
})

log_odds = (
    -3.0
    + 0.05 * X_named["support_calls"]
    + 0.04 * X_named["days_since_login"]
    + 1.50 * X_named["payment_failures"]
    - 0.30 * X_named["satisfaction_score"]
    - 0.02 * X_named["tenure_months"]
    + 0.01 * (X_named["contract_months"] == 1).astype(int)
)
churn_prob = 1 / (1 + np.exp(-log_odds))
y_named = (np.random.rand(n) < churn_prob).astype(int)

X_tr, X_te, y_tr, y_te = train_test_split(
    X_named, y_named, test_size=0.2, random_state=42, stratify=y_named
)

rf_named = RandomForestClassifier(n_estimators=300, n_jobs=-1, random_state=42)
rf_named.fit(X_tr, y_tr)

y_proba = rf_named.predict_proba(X_te)[:, 1]
print(f"ROC-AUC: {roc_auc_score(y_te, y_proba):.4f}")

# Built-in impurity importance
impurity_df = pd.DataFrame({
    "feature": X_named.columns,
    "importance": rf_named.feature_importances_
}).sort_values("importance", ascending=False)

print("\nTop 5 features by impurity importance:")
for _, row in impurity_df.head(5).iterrows():
    print(f"  {row['feature']}: {row['importance']:.4f}")

# Permutation importance (more reliable, unbiased)
perm_result = permutation_importance(rf_named, X_te, y_te, n_repeats=20, random_state=42)
perm_df = pd.DataFrame({
    "feature": X_named.columns,
    "importance_mean": perm_result.importances_mean,
    "importance_std": perm_result.importances_std
}).sort_values("importance_mean", ascending=False)

print("\nTop 5 features by permutation importance:")
for _, row in perm_df.head(5).iterrows():
    print(f"  {row['feature']}: {row['importance_mean']:.4f} ± {row['importance_std']:.4f}")

# Plot
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

top10_impurity = impurity_df.head(10)
axes[0].barh(top10_impurity["feature"][::-1], top10_impurity["importance"][::-1], color="teal")
axes[0].set_title("Impurity Importance (built-in)")
axes[0].set_xlabel("Mean impurity decrease")

top10_perm = perm_df.head(10)
axes[1].barh(top10_perm["feature"][::-1], top10_perm["importance_mean"][::-1], color="teal",
             xerr=top10_perm["importance_std"][::-1], capsize=4)
axes[1].set_title("Permutation Importance")
axes[1].set_xlabel("Mean accuracy decrease")

plt.tight_layout()
plt.savefig("feature_importance_comparison.png", dpi=150)

# Business interpretation
print("""
Business interpretation:
Payment failures are the strongest predictor of churn in both importance measures.
This suggests billing friction is driving customers out before they consciously
decide to cancel — a reactive problem that proactive dunning could address.
Low satisfaction scores compound payment failure risk, and high days_since_login
indicates disengagement that precedes eventual cancellation. Retention teams should
prioritise outreach to customers who have had any payment failure in the last 90 days,
especially those with satisfaction scores of 1 or 2.
""")

Exercise 5 — Stretch: Business Interpretation of Classification Metrics

For each of the following scenarios, answer three questions:

  1. Which metric (accuracy, precision, recall, F1, ROC-AUC, PR-AUC) should be the primary optimisation target?
  2. Which type of error (false positive or false negative) is more costly?
  3. Should you lower or raise the decision threshold from 0.5, and why?

Scenario A — Fraud Detection
A bank processes 1 million transactions per day. Fraud rate: 0.05%. Every missed fraud costs the bank $500. Every false alarm requires a $10 analyst review.

Scenario B — Tumour Classification
A radiologist's AI assistant screens chest X-rays for lung nodules. A missed nodule means a patient goes undiagnosed for months. A false alarm means an unnecessary follow-up CT scan.

Scenario C — Spam Filter
A corporate email system classifies inbound emails. A false alarm sends a legitimate client email to the junk folder. A missed spam email is mildly annoying.

Scenario D — Job Application Screening
An HR system screens 10,000 applications down to 200 for human review. A missed good candidate is a lost hire. A false positive wastes a recruiter's time.

Show answers

Scenario A — Fraud Detection

  • Primary metric: PR-AUC (because fraud rate is very low — 0.05% — ROC-AUC is inflated by the huge number of true negatives)
  • More costly error: False negative (missed fraud = $500 loss vs $10 for a false alarm review)
  • Threshold: Lower below 0.5 to increase recall. The math: missing fraud costs 50x more than reviewing a false alarm, so the model should flag more aggressively.

Scenario B — Tumour Classification

  • Primary metric: Recall (maximise sensitivity). Also report F2-score (weights recall twice as heavily as precision)
  • More costly error: False negative — missing a nodule delays diagnosis and treatment, potentially fatally
  • Threshold: Lower significantly (perhaps 0.2–0.3) to catch almost every true positive, accepting many false alarms. The radiologist reviews all flagged cases anyway, so false alarms increase workload but do not harm patients.

Scenario C — Spam Filter

  • Primary metric: Precision — of what we label spam, how many actually are spam?
  • More costly error: False positive — a legitimate client email in junk is a relationship risk
  • Threshold: Raise above 0.5 (perhaps 0.7–0.8) to only flag high-confidence spam. Some spam will reach the inbox, but client communications will not be missed.

Scenario D — Job Application Screening

  • Primary metric: Recall within the budget constraint. You need to pass the best 200 candidates through — this is a ranking problem. Use ROC-AUC to evaluate ranking quality, then set the threshold to output exactly 200 positives (top-k selection)
  • More costly error: False negative — missing a strong candidate is a direct hiring loss
  • Threshold: Do not use a fixed threshold. Instead, rank all applicants by predicted probability and take the top 200. This is the correct framing for a capacity-constrained selection problem.

Previous: Classification Metrics | Next: Clustering Techniques