Evaluation and Interpretation

Training a model is the easy part. Knowing whether it is actually useful — and communicating that to people who did not build it — is the hard part. This file walks through every layer of evaluation: confusion matrix translated into business language, ROC and PR curves, feature importance, threshold analysis, and a ROI framing your stakeholders will understand.

Learning Objectives

By the end of this file you will be able to:

• Read a confusion matrix and translate each cell into a business outcome
• Interpret precision, recall, and F1 for each class correctly
• Explain the difference between ROC-AUC and PR-AUC and when each is more informative
• Identify the top features driving predictions
• Build a threshold analysis table that helps a team pick the right operating point
• Calculate the business ROI of model-guided targeting vs. random targeting
• Describe what a lift curve shows and how to read it

Setup

This file continues from model-building. The gradient boosting pipeline (gb_pipeline) fitted at threshold 0.35 is the best model from that file. Run all prior setup code, then paste this code beneath it.

from sklearn.metrics import (
    confusion_matrix, classification_report,
    roc_curve, auc, precision_recall_curve,
    ConfusionMatrixDisplay
)
import matplotlib.pyplot as plt
import matplotlib.ticker as mticker
import numpy as np
import pandas as pd
from sklearn.utils.class_weight import compute_sample_weight

# Refit best model on training data
gb_pipeline.fit(X_train_eng, y_train,
                classifier__sample_weight=compute_sample_weight("balanced", y_train))

# Probabilities on test set
gb_proba = gb_pipeline.predict_proba(X_test_eng)[:, 1]

# Predictions at chosen threshold
THRESHOLD = 0.35
gb_preds  = (gb_proba >= THRESHOLD).astype(int)

Step 1: Confusion Matrix

The confusion matrix is 2x2, but each cell has a concrete business meaning.

cm = confusion_matrix(y_test, gb_preds)

fig, ax = plt.subplots(figsize=(6, 5))
disp = ConfusionMatrixDisplay(confusion_matrix=cm,
                               display_labels=["Retained", "Churned"])
disp.plot(ax=ax, colorbar=False, cmap="Blues")
ax.set_title("Confusion Matrix — Gradient Boosting (threshold = 0.35)", pad=12)
plt.tight_layout()
plt.show()

tn, fp, fn, tp = cm.ravel()
print(f"\nTrue Negatives  (TN): {tn} — retained customers correctly left alone")
print(f"False Positives (FP): {fp} — loyal customers who got an unnecessary call")
print(f"False Negatives (FN): {fn} — churners the model missed entirely")
print(f"True Positives  (TP): {tp} — churners the model correctly flagged")

Business Translation

Cell	Count	What it means for the retention team
True Positive	~52	Customers about to leave who received a timely retention call. Revenue saved.
False Positive	~43	Loyal customers who received an unnecessary call. Minor cost — one call each.
False Negative	~7	Churners the model missed. They will cancel without any intervention.
True Negative	~98	Retained customers correctly left alone. No wasted resources.

Success

At threshold 0.35 the model catches ~88% of churners (52 out of 59) while generating 43 unnecessary calls. If each retention call costs 15 minutes of agent time, false positives cost 43 × 15 = 645 minutes ≈ 11 agent-hours per test cycle. That is a manageable cost compared to losing 52 customers.

Step 2: Classification Report

print(classification_report(y_test, gb_preds,
                             target_names=["Retained", "Churned"],
                             digits=3))

Expected output:

              precision    recall  f1-score   support

    Retained      0.934     0.695     0.797       141
     Churned      0.548     0.881     0.676        59

    accuracy                          0.750       200
   macro avg      0.741     0.788     0.737       200
weighted avg      0.817     0.750     0.763       200

Reading the report:

• Retained class: precision 0.93 means 93% of customers the model predicted as "retained" are truly retained. Recall 0.695 means the model correctly identifies 69.5% of all truly retained customers.
• Churn class: recall 0.88 is the number that matters most. The model catches 88% of churners. Precision 0.55 means 55% of everyone the model flags will actually churn.
• Macro avg: unweighted average across both classes. More informative than accuracy for imbalanced datasets.

Info

The accuracy of 75% looks lower than the dummy's 70%, but this model catches 52 churners vs. zero. Never use accuracy as the primary metric on an imbalanced classification problem.

Step 3: ROC Curve vs. PR Curve

Two curves — different questions.

ROC curve asks: across all possible thresholds, how does my true positive rate compare to my false positive rate?

PR curve asks: across all possible thresholds, how does my precision compare to my recall?

For imbalanced datasets, the PR curve is more informative because it focuses entirely on the minority class (churners) rather than averaging performance across both classes.

fig, axes = plt.subplots(1, 2, figsize=(13, 5))

# --- ROC Curve ---
fpr, tpr, roc_thresholds = roc_curve(y_test, gb_proba)
roc_auc_val = auc(fpr, tpr)

axes[0].plot(fpr, tpr, color="#0D9488", linewidth=2.5,
             label=f"Gradient Boosting (AUC = {roc_auc_val:.3f})")
axes[0].plot([0, 1], [0, 1], "k--", linewidth=1, label="Random baseline")
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate (Recall)")
axes[0].set_title("ROC Curve")
axes[0].legend()

# --- PR Curve ---
precisions, recalls, pr_thresholds = precision_recall_curve(y_test, gb_proba)
pr_auc_val = auc(recalls, precisions)
baseline_precision = y_test.mean()

axes[1].plot(recalls, precisions, color="#F59E0B", linewidth=2.5,
             label=f"Gradient Boosting (AUC = {pr_auc_val:.3f})")
axes[1].axhline(y=baseline_precision, color="#94A3B8", linestyle="--", linewidth=1,
                label=f"Random baseline ({baseline_precision:.2f})")
axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_title("Precision-Recall Curve")
axes[1].legend()

plt.suptitle("ROC vs Precision-Recall Curve Comparison", fontsize=13, y=1.01)
plt.tight_layout()
plt.show()

print(f"AUC-ROC:          {roc_auc_val:.3f}")
print(f"AUC-PR:           {pr_auc_val:.3f}")
print(f"Random PR baseline: {baseline_precision:.3f}")
# A model with AUC-PR >> 0.30 is doing meaningful work

Info

For a dataset with 30% positive rate, a random classifier achieves AUC-PR of ~0.30. Your model should achieve AUC-PR of ~0.65–0.75. That gap represents the model's real predictive value. The ROC curve's random baseline is always 0.50 regardless of class imbalance, which is why ROC can be misleading here.

Step 4: Feature Importance

Which features drove the model's predictions? For gradient boosting, feature_importances_ gives the mean impurity decrease across all trees for each feature.

# Get feature names from ColumnTransformer
def get_feature_names(preprocessor, onehot_features):
    """Extract ordered feature names from a fitted ColumnTransformer."""
    ohe = preprocessor.named_transformers_["onehot"]["encoder"]
    onehot_names = ohe.get_feature_names_out(onehot_features).tolist()
    return (
        ["age", "tenure_months", "support_calls", "monthly_fee",
         "calls_per_tenure", "fee_per_product"]      # numeric
        + ["has_contract", "high_call_flag"]           # binary passthrough
        + onehot_names                                 # one-hot
        + ["tenure_bucket"]                            # ordinal
    )

feature_names = get_feature_names(preprocessor, ["segment", "region"])
importances   = gb_pipeline.named_steps["classifier"].feature_importances_

feat_imp = pd.Series(importances, index=feature_names).sort_values(ascending=False)
top10 = feat_imp.head(10)

fig, ax = plt.subplots(figsize=(9, 5))
bars = ax.barh(top10.index[::-1], top10.values[::-1], color="#0D9488", edgecolor="white")
ax.set_xlabel("Feature Importance (Mean Impurity Decrease)")
ax.set_title("Top 10 Feature Importances — Gradient Boosting")
plt.tight_layout()
plt.show()

print(top10.round(4))

Expected top features:

Rank	Feature	Interpretation
1	calls_per_tenure	Contact intensity — the dominant signal
2	tenure_months	How long they have been a customer
3	has_contract	Contract status creates strong retention friction
4	high_call_flag	Binary threshold feature capturing crisis-level friction
5	segment_Basic	Basic plan customers churn at ~40%
6	monthly_fee	Higher fees correlate with Enterprise (low churn)
7	fee_per_product	Value density signal
8	tenure_bucket	Lifecycle stage (ordinal)
9	support_calls	Raw call count (captured partly by engineered features)
10	segment_Enterprise	Negative churn risk

Success

The feature importance ranking confirms every EDA finding: tenure and support interactions dominate, segment and contract add lift, and region features (region_North, region_South, etc.) will appear near the bottom with near-zero importance. This is validation that the model learned real patterns, not noise.

Step 5: Threshold Analysis Table

Help your team pick the right operating point by showing the tradeoff at five thresholds.

thresholds_to_check = [0.30, 0.35, 0.40, 0.45, 0.50]
rows = []

for t in thresholds_to_check:
    preds = (gb_proba >= t).astype(int)
    tp = ((preds == 1) & (y_test == 1)).sum()
    fp = ((preds == 1) & (y_test == 0)).sum()
    fn = ((preds == 0) & (y_test == 1)).sum()
    prec = precision_score(y_test, preds, zero_division=0)
    rec  = recall_score(y_test, preds)
    f1   = f1_score(y_test, preds)
    flagged_per_1000 = int((preds.sum() / len(y_test)) * 1000)
    rows.append({
        "Threshold":        t,
        "Precision":        round(prec, 3),
        "Recall":           round(rec, 3),
        "F1":               round(f1, 3),
        "Churners caught":  tp,
        "Churners missed":  fn,
        "False calls":      fp,
        "Flagged per 1,000": flagged_per_1000,
    })

threshold_df = pd.DataFrame(rows).set_index("Threshold")
print(threshold_df.to_string())

Expected output (approximate):

Threshold	Precision	Recall	F1	Churners caught	Churners missed	False calls	Flagged per 1,000
0.30	0.51	0.93	0.66	55	4	53	540
0.35	0.55	0.88	0.67	52	7	43	475
0.40	0.61	0.83	0.70	49	10	31	400
0.45	0.66	0.78	0.71	46	13	24	350
0.50	0.70	0.73	0.71	43	16	18	305

How to read this table with your team: "If the retention team can handle 150 calls per 1,000 customers per week, use threshold 0.50 — it flags ~305 customers per 1,000 scaled to our full base, catching 73% of churners. If you can handle 200 calls, drop to 0.35 and catch 88%."

Tip

Print this table in the stakeholder presentation. Replace "per 1,000" with the actual customer count. Stakeholders make better decisions when the tradeoff is framed in terms of calls and customers, not precision and recall.

Step 6: Business ROI Framing

Quantify the model's value in terms the business cares about.

# Assumptions
total_customers      = 10_000        # full customer base
weekly_call_capacity = 150           # retention team can call 150/week
churn_rate           = 0.30          # 30% churn rate = 3,000 churners
avg_customer_ltv     = 24_000        # INR — 24 months × avg monthly fee

# Random targeting (no model)
random_precision = churn_rate         # if you call 150 at random, ~30% are churners
churners_saved_random = int(weekly_call_capacity * random_precision)

# Model-guided targeting at threshold 0.45 (precision ~0.66)
model_precision = 0.66
churners_saved_model = int(weekly_call_capacity * model_precision)

# Difference
extra_churners_saved = churners_saved_model - churners_saved_random
revenue_saved_per_week = extra_churners_saved * avg_customer_ltv

print(f"Random targeting saves:  {churners_saved_random} churners/week")
print(f"Model targeting saves:   {churners_saved_model} churners/week")
print(f"Extra churners retained: {extra_churners_saved}/week")
print(f"Revenue saved per week:  INR {revenue_saved_per_week:,.0f}")
print(f"Revenue saved per year:  INR {revenue_saved_per_week * 52:,.0f}")

# Expected output:
# Random targeting saves:  45 churners/week
# Model targeting saves:   99 churners/week
# Extra churners retained: 54/week
# Revenue saved per week:  INR 12,96,000
# Revenue saved per year:  INR 6,73,92,000

Success

This calculation is always approximate — it assumes a 100% save rate on every retention call, which is optimistic. A realistic save rate is 20–40%. Adjust accordingly: at 30% save rate, the model still saves 16 additional churners per week vs. random targeting. Run the number both ways and present both to stakeholders.

Step 7: Lift Curve

A lift curve answers: "How much better is model-guided targeting than random guessing in the top X% of predictions?"

# Sort test set by predicted churn probability, descending
lift_df = pd.DataFrame({
    "proba":    gb_proba,
    "actual":   y_test.values,
}).sort_values("proba", ascending=False).reset_index(drop=True)

# Cumulative churn captured as a fraction of all churners
total_churners = lift_df["actual"].sum()
lift_df["cumulative_churners"] = lift_df["actual"].cumsum()
lift_df["cumulative_pct_pop"]  = (lift_df.index + 1) / len(lift_df)
lift_df["cumulative_recall"]   = lift_df["cumulative_churners"] / total_churners
lift_df["lift"]                = lift_df["cumulative_recall"] / lift_df["cumulative_pct_pop"]

# Plot
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(lift_df["cumulative_pct_pop"] * 100,
        lift_df["cumulative_recall"] * 100,
        color="#0D9488", linewidth=2.5, label="Gradient Boosting")
ax.plot([0, 100], [0, 100], "k--", linewidth=1, label="Random baseline")
ax.fill_between(lift_df["cumulative_pct_pop"] * 100,
                lift_df["cumulative_recall"] * 100,
                lift_df["cumulative_pct_pop"] * 100,
                alpha=0.1, color="#0D9488")
ax.set_xlabel("% of Customers Contacted (sorted by score)")
ax.set_ylabel("% of Churners Captured")
ax.set_title("Cumulative Gains / Lift Curve")
ax.legend()
ax.set_xlim(0, 100)
ax.set_ylim(0, 100)
plt.tight_layout()
plt.show()

# Print top decile lift
top_decile = lift_df[lift_df["cumulative_pct_pop"] <= 0.10].iloc[-1]
print(f"Top 10% of model scores captures {top_decile['cumulative_recall']:.1%} of churners")
print(f"Lift in top decile: {top_decile['lift']:.2f}x vs random")
# Expected: ~55–65% of churners captured in top 10%; lift ~5.5–6.5x

A lift of 5.5x in the top decile means: if you contact the 10% of customers the model ranks highest, you will reach 5.5x more churners than if you had picked 10% at random.

Step 8: What Would You Do Next

This model is a strong starting point. In a production context, the next steps would be:

SHAP for individual explanations. The feature importance table shows global patterns. SHAP (SHapley Additive exPlanations) explains why the model flagged a specific customer — essential for the retention team to personalise their call. "This customer is flagged because their calls_per_tenure is 0.8 and they have no contract."

# pip install shap
import shap
explainer  = shap.TreeExplainer(gb_pipeline.named_steps["classifier"])
X_test_transformed = preprocessor.transform(X_test_eng)
shap_values = explainer.shap_values(X_test_transformed)
shap.summary_plot(shap_values, X_test_transformed, feature_names=feature_names)

Survival analysis (time-to-churn). The current model predicts whether a customer will churn in the next 30 days. Survival analysis (Kaplan-Meier, Cox proportional hazards) models when they will churn — more useful for long-term retention planning.

A/B test the retention intervention. The model is only valuable if the retention calls actually work. Set up an A/B test: randomly assign high-risk customers to "called" vs. "not called" and measure whether the intervention reduces churn. Without this test, you cannot attribute churn reduction to the model.

Monitor for drift. Customer behaviour changes over time. Set up a monthly check: recompute AUC-ROC on new data. If AUC drops more than 5 percentage points from the deployment baseline, retrain the model on more recent data.

Tip

The order of impact for production improvements: (1) A/B test first — prove the intervention works; (2) SHAP for call personalisation; (3) monthly monitoring pipeline; (4) survival analysis for longer-horizon planning. Do not skip to step 4 before step 1.

---

← Model Building | Next: Interview Questions →