Evaluation and Report — Churn Prediction

This is the one moment in the project where you look at the test set. Do it once, record the numbers, and do not go back and retune based on what you see. If you retune after seeing test results, those results are no longer valid.

By the end of this phase you have the numbers for your report and a written results section ready to share with a non-technical stakeholder.

---

Step 1 — Final Test Set Evaluation

from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
    f1_score,
    ConfusionMatrixDisplay,
    RocCurveDisplay,
)
import matplotlib.pyplot as plt

# best_model is the fitted Pipeline from 04-modeling-pipeline.md
y_pred  = best_model.predict(X_test)
y_proba = best_model.predict_proba(X_test)[:, 1]   # probability of churn

print("=== Classification Report ===")
print(classification_report(y_test, y_pred, target_names=["Retained", "Churned"]))

# Output (approximate):
# === Classification Report ===
#               precision    recall  f1-score   support
#
#     Retained       0.89      0.88      0.89       141
#      Churned       0.73      0.75      0.74        59
#
#     accuracy                           0.84       200
#    macro avg       0.81      0.82      0.81       200
# weighted avg       0.84      0.84      0.84       200

The number that matters for this project is F1 for "Churned": 0.74. That exceeds the minimum bar of 0.60 set in the project brief.

Success

Test set F1 (churn class) = 0.74. Cross-validation estimated 0.714. The difference is small, which means the model generalised as expected and the CV estimate was honest. Record this in your report.

---

Step 2 — Confusion Matrix

fig, ax = plt.subplots(figsize=(5, 4))
ConfusionMatrixDisplay.from_predictions(
    y_test, y_pred,
    display_labels=["Retained", "Churned"],
    cmap="Blues",
    ax=ax,
)
ax.set_title("Confusion Matrix — Test Set")
plt.tight_layout()
plt.show()

How to read the confusion matrix for this business problem:

	Predicted: Retained	Predicted: Churned
Actual: Retained	True Negatives (TN)	False Positives (FP) — wasted retention spend
Actual: Churned	False Negatives (FN) — missed churners	True Positives (TP) — correctly flagged

• False Negatives are the expensive error. A customer who churns without being flagged generates zero retention opportunity.
• False Positives are the cheaper error. The retention team contacts a customer who was going to stay anyway — some wasted cost, but no revenue loss.

Info

In most churn programmes, businesses are willing to accept 2–3 false positives for every true positive because the cost of a retention offer ($10–20) is far less than the lifetime value of the customer. This is the business case for adjusting the classification threshold.

---

Step 3 — ROC Curve

fig, ax = plt.subplots(figsize=(6, 5))
RocCurveDisplay.from_predictions(y_test, y_proba, ax=ax, name="Gradient Boosting")
ax.plot([0, 1], [0, 1], "k--", label="Random baseline")
ax.set_title("ROC Curve — Test Set")
ax.legend()
plt.tight_layout()
plt.show()

roc_auc = roc_auc_score(y_test, y_proba)
print(f"ROC-AUC: {roc_auc:.3f}")
# Output: ROC-AUC: 0.881  (approximate)

ROC-AUC of 0.88 means the model ranks a randomly selected churner above a randomly selected non-churner 88% of the time. This is a strong result for a dataset of 1,000 rows.

---

Step 4 — Feature Importance

import pandas as pd
import numpy as np

# Extract the fitted classifier from the pipeline
clf = best_model.named_steps["classifier"]

# Get feature names from the preprocessor
feature_names = best_model.named_steps["preprocessor"].get_feature_names_out()

# GradientBoostingClassifier uses feature_importances_
importances = clf.feature_importances_
importance_df = (
    pd.DataFrame({"feature": feature_names, "importance": importances})
    .sort_values("importance", ascending=False)
    .reset_index(drop=True)
)

print(importance_df.head(10).to_string(index=False))

# Output (approximate — top features):
#                            feature  importance
#  cat__contract_type_Month-to-Month    0.218
#             num__calls_per_tenure    0.187
#              num__tenure_months     0.142
#              num__support_calls     0.119
#  cat__contract_type_Two Year        0.089
#       num__charges_per_product      0.071
#         num__monthly_charges        0.058
#         num__has_tech_support        0.044
#          num__num_products          0.038
#  cat__payment_method_Electronic...  0.034

# Horizontal bar chart
fig, ax = plt.subplots(figsize=(8, 6))
importance_df.head(10).plot.barh(
    x="feature", y="importance", ax=ax, legend=False, color="steelblue"
)
ax.invert_yaxis()
ax.set_xlabel("Feature Importance")
ax.set_title("Top 10 Features — Gradient Boosting")
plt.tight_layout()
plt.show()

Findings: The top four features — contract_type_Month-to-Month, calls_per_tenure, tenure_months, and support_calls — account for ~67% of total importance. This confirms the EDA hypotheses. The two engineered features (calls_per_tenure, charges_per_product) rank higher than their source columns individually, which validates the feature engineering step.

Success

If your feature importance results agree with your EDA findings (contract type and support calls are the dominant predictors), your model is learning real signal, not noise. If they disagree dramatically, investigate before reporting.

---

Step 5 — Threshold Adjustment for Recall

The default threshold is 0.50 — predict churn if P(churn) > 0.50. But the business may prefer a lower threshold to catch more churners, accepting more false positives.

from sklearn.metrics import precision_recall_curve
import numpy as np

thresholds = np.arange(0.20, 0.80, 0.05)

print(f"{'Threshold':>10} {'Precision':>10} {'Recall':>8} {'F1':>8}")
print("-" * 42)
for thresh in thresholds:
    preds = (y_proba >= thresh).astype(int)
    p = precision_score(y_test, preds, zero_division=0)
    r = recall_score(y_test, preds)
    f = f1_score(y_test, preds, zero_division=0)
    print(f"{thresh:>10.2f} {p:>10.3f} {r:>8.3f} {f:>8.3f}")

# Output (approximate):
#  Threshold  Precision   Recall       F1
# ------------------------------------------
#       0.20      0.463    0.966    0.627
#       0.25      0.494    0.949    0.650
#       0.30      0.533    0.932    0.679
#       0.35      0.582    0.898    0.706
#       0.40      0.634    0.864    0.732
#       0.45      0.688    0.814    0.746
#       0.50      0.729    0.746    0.737
#       0.55      0.771    0.712    0.740
#       0.60      0.806    0.678    0.737
#       0.65      0.840    0.627    0.718
#       0.70      0.878    0.559    0.683

from sklearn.metrics import precision_score, recall_score

Tip

A threshold of 0.35–0.40 may be the better business choice here: recall goes from 0.75 to 0.86–0.90, meaning you catch 11–15 additional churners per 100, at the cost of 10–15 additional false positives (extra retention contacts). Most retention teams will take that trade. Present this table to stakeholders and let them pick the operating point.

---

Step 6 — Write the Results Section

This is what actually gets read. Executives, product managers, and interviewers read the results section; they skim everything else.

Fill in this template with your actual numbers:

## Results

**Model:** Gradient Boosting Classifier (tuned)
**Test set F1 (churn class):** 0.74
**Test set ROC-AUC:** 0.88
**Test set accuracy:** 84%

At the default threshold (0.50), the model correctly identifies 75% of customers who will
churn (recall = 0.75) with a precision of 73%, meaning 73 of every 100 flagged customers
are genuine churners.

**Key drivers of churn identified:**
1. Month-to-Month contracts — customers on month-to-month contracts are 8× more likely to
   churn than Two Year contract holders.
2. High support call rate relative to tenure — a new customer who calls support 4 times in
   their first 3 months has a predicted churn probability of over 60%.
3. Short tenure — the highest-risk period is the first 12 months of a customer's lifecycle.

**Business recommendation:**
Deploy this model with a threshold of 0.35 to maximise recall. At this threshold, the model
flags approximately 90% of future churners 2–4 weeks before churn, at a false positive rate
of ~45%. Given an average customer lifetime value of $[X], the expected return on a
$15 retention offer to flagged customers is positive if the offer converts at least [Y]%.

**Limitations:**
- Dataset is synthetic. Model should be retrained on real customer data before deployment.
- No temporal features (recency of last interaction, trend in call frequency). Adding these
  would likely increase recall further.
- The model does not account for voluntary vs. involuntary churn (payment failure). Separating
  these would allow more targeted interventions.

---

Step 7 — Presenting to a Non-Technical Audience

Three rules for presenting model results to stakeholders who do not know what F1 means:

1. Translate to business quantities. Do not say "recall of 0.75". Say "for every 100 customers who will churn, our model identifies 75 of them before they leave."

2. Frame the error in business cost. "The 25 customers we miss represent approximately $[revenue] in annual lost revenue. The 27 customers we flag who won't churn receive an unnecessary $15 retention offer — a total cost of $405."

3. Show the threshold tradeoff as a business decision. Present the precision-recall table from Step 5 and ask stakeholders: "How many extra contacts are you willing to make to catch one additional churner?" That is the operating point decision — let them make it.

Warning

Never let a stakeholder walk away thinking your model is "84% accurate." Accuracy is meaningless for imbalanced classification. If they anchor on that number, the next model comparison will be unfair and you will lose credibility when the model "performs worse" on a dataset with a different base rate.

---

04-modeling-pipeline | 06-submission-checklist