Model Building¶
You have clean, engineered features and a preprocessor pipeline. Now you turn that work into a classifier. This file walks from a naive baseline through three real models, explains why recall is the metric that matters, and shows how to tune the decision threshold to match the business constraint.
Learning Objectives¶
By the end of this file you will be able to:
- Establish a meaningful baseline using
DummyClassifier - Train logistic regression, random forest, and gradient boosting classifiers with class imbalance handling
- Compare models using a multi-metric table
- Adjust the decision threshold from 0.5 to a value that improves recall
- Validate the best model with stratified cross-validation
Setup¶
This file continues directly from feature-engineering. Run the dataset generation, the add_engineered_features function, and the preprocessor definition from that file first, then paste this code beneath it.
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import (
accuracy_score, precision_score, recall_score,
f1_score, roc_auc_score, classification_report,
precision_recall_curve, roc_curve
)
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings("ignore")
def evaluate(model, X_tr, y_tr, X_te, y_te, threshold=0.5):
"""Fit model and return a dict of evaluation metrics on the test set."""
model.fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]
preds = (proba >= threshold).astype(int)
return {
"accuracy": accuracy_score(y_te, preds),
"precision": precision_score(y_te, preds, zero_division=0),
"recall": recall_score(y_te, preds),
"f1": f1_score(y_te, preds),
"auc_roc": roc_auc_score(y_te, proba),
}
Step 1: Baseline — DummyClassifier¶
Before training any real model, establish what "doing nothing useful" looks like. The DummyClassifier with strategy='most_frequent' always predicts the majority class — in this case, "not churned".
dummy = Pipeline([
("preprocessor", preprocessor),
("classifier", DummyClassifier(strategy="most_frequent")),
])
dummy.fit(X_train_eng, y_train)
dummy_preds = dummy.predict(X_test_eng)
print("Dummy Classifier:")
print(f" Accuracy: {accuracy_score(y_test, dummy_preds):.1%}")
print(f" Recall: {recall_score(y_test, dummy_preds):.1%}")
print(f" Precision: {precision_score(y_test, dummy_preds, zero_division=0):.1%}")
# Expected output:
# Accuracy: ~70%
# Recall: 0% <- catches zero churners
# Precision: 0% <- no positive predictions at all
Warning
70% accuracy with 0% recall is not a model — it is a failure mode that looks like success on the wrong metric. A retention team using this "model" would call zero customers and lose every churner. This is your floor, not your target.
Step 2: Logistic Regression¶
Logistic regression is fast, interpretable, and provides well-calibrated probabilities. Start here.
class_weight='balanced' automatically adjusts the loss function to penalise misclassification of the minority class (churners) more heavily. It is equivalent to upsampling the minority class without duplicating rows.
lr_pipeline = Pipeline([
("preprocessor", preprocessor),
("classifier", LogisticRegression(
class_weight="balanced",
max_iter=1000,
random_state=42,
)),
])
lr_pipeline.fit(X_train_eng, y_train)
lr_proba = lr_pipeline.predict_proba(X_test_eng)[:, 1]
lr_preds = (lr_proba >= 0.5).astype(int)
print(classification_report(y_test, lr_preds, target_names=["Retained", "Churned"]))
Expected report:
precision recall f1-score support
Retained 0.88 0.78 0.83 141
Churned 0.61 0.76 0.68 59
accuracy 0.77 200
macro avg 0.75 0.77 0.76 200
weighted avg 0.80 0.77 0.78 200
Info
Logistic regression with class_weight='balanced' trades some precision for recall. You will see it flag more customers as at-risk (some of whom are fine), but it misses fewer actual churners. That is the correct tradeoff for this business problem.
Step 3: Random Forest¶
Random forest is an ensemble of decision trees trained on bootstrap samples. It handles non-linear relationships and feature interactions well without manual scaling — though scaling in the pipeline does not hurt it.
rf_pipeline = Pipeline([
("preprocessor", preprocessor),
("classifier", RandomForestClassifier(
n_estimators=200,
class_weight="balanced",
max_depth=8,
min_samples_leaf=5,
random_state=42,
n_jobs=-1,
)),
])
rf_pipeline.fit(X_train_eng, y_train)
rf_proba = rf_pipeline.predict_proba(X_test_eng)[:, 1]
rf_preds = (rf_proba >= 0.5).astype(int)
print(classification_report(y_test, rf_preds, target_names=["Retained", "Churned"]))
Step 4: Gradient Boosting¶
Gradient boosting builds trees sequentially — each tree corrects the residual errors of the previous one. It typically outperforms random forest on structured tabular data when tuned properly.
GradientBoostingClassifier does not have a class_weight parameter. Handle imbalance instead with sample_weight or by adjusting the subsample ratio. Here we use the simpler approach: pass sample_weight to the fit call.
from sklearn.utils.class_weight import compute_sample_weight
gb_pipeline = Pipeline([
("preprocessor", preprocessor),
("classifier", GradientBoostingClassifier(
n_estimators=200,
learning_rate=0.05,
max_depth=4,
subsample=0.8,
random_state=42,
)),
])
sample_weights = compute_sample_weight("balanced", y_train)
gb_pipeline.fit(X_train_eng, y_train,
classifier__sample_weight=sample_weights)
gb_proba = gb_pipeline.predict_proba(X_test_eng)[:, 1]
gb_preds = (gb_proba >= 0.5).astype(int)
print(classification_report(y_test, gb_preds, target_names=["Retained", "Churned"]))
Tip
The classifier__sample_weight syntax is how you pass fit-time parameters to a named step inside a Pipeline. The double underscore (__) separates the step name (classifier) from the parameter name (sample_weight).
Step 5: Model Comparison¶
Collect all results into a single table so you can compare models side by side.
results = {}
results["Dummy"] = evaluate(dummy, X_train_eng, y_train, X_test_eng, y_test)
results["Logistic Regression"] = evaluate(lr_pipeline, X_train_eng, y_train, X_test_eng, y_test)
results["Random Forest"] = evaluate(rf_pipeline, X_train_eng, y_train, X_test_eng, y_test)
results["Gradient Boosting"] = evaluate(gb_pipeline, X_train_eng, y_train, X_test_eng, y_test,
# GB needs sample_weight — refit manually for the dict
)
# Note: for GB, refit manually with sample_weight before calling evaluate,
# or reuse the already-fitted pipeline and call predict_proba directly.
comparison = pd.DataFrame(results).T.round(3)
comparison.columns = ["Accuracy", "Precision (Churn)", "Recall (Churn)", "F1 (Churn)", "AUC-ROC"]
print(comparison.sort_values("Recall (Churn)", ascending=False))
Expected comparison (approximate):
| Model | Accuracy | Precision (Churn) | Recall (Churn) | F1 (Churn) | AUC-ROC |
|---|---|---|---|---|---|
| Gradient Boosting | 0.80 | 0.65 | 0.78 | 0.71 | 0.87 |
| Random Forest | 0.79 | 0.63 | 0.76 | 0.69 | 0.85 |
| Logistic Regression | 0.77 | 0.61 | 0.76 | 0.68 | 0.83 |
| Dummy | 0.71 | 0.00 | 0.00 | 0.00 | 0.50 |
Step 6: Why Recall Matters Here¶
The retention team can make roughly 150 calls per week. Every churner they miss costs the business the full remaining lifetime value of that customer. Every false positive costs one unnecessary 15-minute call.
The asymmetry is stark:
- Missing a churner (false negative): lose customer permanently; expected loss = months of remaining revenue
- Calling a loyal customer (false positive): minor inconvenience; cost = 15 minutes of agent time
This means you should optimise for recall on the churn class, not accuracy, and not precision. A model with 76% recall and 63% precision is dramatically more valuable to the business than a model with 85% precision and 45% recall.
Success
The right metric is determined by the business cost structure, not by convention. Spend five minutes mapping out false negative cost vs. false positive cost before you write any model code. That conversation with your stakeholder is the most important modelling decision you make.
Step 7: Threshold Tuning¶
The default decision threshold of 0.5 was not designed for imbalanced datasets with asymmetric costs. Lowering the threshold means the model flags more customers as at-risk, which increases recall at the cost of precision.
Plot the Precision-Recall Curve¶
# Use the best model — gradient boosting
gb_pipeline.fit(X_train_eng, y_train,
classifier__sample_weight=compute_sample_weight("balanced", y_train))
gb_proba = gb_pipeline.predict_proba(X_test_eng)[:, 1]
precisions, recalls, thresholds = precision_recall_curve(y_test, gb_proba)
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(thresholds, precisions[:-1], label="Precision", color="#0D9488", linewidth=2)
ax.plot(thresholds, recalls[:-1], label="Recall", color="#F59E0B", linewidth=2)
# Mark threshold = 0.5
idx_50 = np.searchsorted(thresholds, 0.5)
ax.axvline(x=0.5, color="#94A3B8", linestyle="--", alpha=0.7, label="threshold = 0.5")
# Mark threshold = 0.35
idx_35 = np.searchsorted(thresholds, 0.35)
ax.axvline(x=0.35, color="#2DD4BF", linestyle="--", alpha=0.7, label="threshold = 0.35")
ax.set_xlabel("Decision Threshold")
ax.set_ylabel("Score")
ax.set_title("Precision and Recall vs Decision Threshold")
ax.legend()
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
plt.tight_layout()
plt.show()
Find the F1-Maximising Threshold¶
# Compute F1 at every threshold
f1_scores = 2 * (precisions[:-1] * recalls[:-1]) / (precisions[:-1] + recalls[:-1] + 1e-9)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]
print(f"Best threshold (max F1): {best_threshold:.3f}")
print(f" Precision at best: {precisions[best_idx]:.3f}")
print(f" Recall at best: {recalls[best_idx]:.3f}")
print(f" F1 at best: {f1_scores[best_idx]:.3f}")
# Expect threshold ~0.35–0.40
Compare Threshold 0.5 vs 0.35¶
for thresh in [0.50, 0.35]:
preds = (gb_proba >= thresh).astype(int)
flagged = preds.sum()
prec = precision_score(y_test, preds)
rec = recall_score(y_test, preds)
f1 = f1_score(y_test, preds)
print(f"Threshold {thresh:.2f} | Precision: {prec:.2f} | Recall: {rec:.2f} | "
f"F1: {f1:.2f} | Flagged: {flagged}/{len(y_test)}")
# Expected output:
# Threshold 0.50 | Precision: 0.65 | Recall: 0.78 | F1: 0.71 | Flagged: ~71
# Threshold 0.35 | Precision: 0.55 | Recall: 0.88 | F1: 0.67 | Flagged: ~95
At threshold 0.35, you catch more churners (88% vs 78%) at the cost of more false positive calls. Whether that tradeoff is worth it depends on the team's call capacity — exactly the conversation to have with the retention manager.
Info
There is no universally correct threshold. The right threshold is the one where the precision-recall tradeoff aligns with the team's operational capacity. If the team can handle 100 calls per week and you have 1,000 customers, a threshold that flags 10% is appropriate. If they can handle 200 calls, you can lower the threshold further to catch more churners.
Step 8: Cross-Validation¶
A single train-test split can produce lucky or unlucky results depending on which customers landed in each fold. Stratified k-fold cross-validation gives a more reliable estimate of model performance.
from sklearn.model_selection import StratifiedKFold, cross_val_score
# Use the full engineered dataset for CV
X_full_eng = add_engineered_features(X)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Rebuild the GB pipeline for CV (sample_weight complicates cross_val_score;
# use class_weight via a workaround or switch to RF for CV demonstration)
rf_cv_pipeline = Pipeline([
("preprocessor", preprocessor),
("classifier", RandomForestClassifier(
n_estimators=200,
class_weight="balanced",
max_depth=8,
min_samples_leaf=5,
random_state=42,
n_jobs=-1,
)),
])
cv_auc = cross_val_score(rf_cv_pipeline, X_full_eng, y,
cv=skf, scoring="roc_auc", n_jobs=-1)
cv_recall = cross_val_score(rf_cv_pipeline, X_full_eng, y,
cv=skf, scoring="recall", n_jobs=-1)
print(f"AUC-ROC: {cv_auc.mean():.3f} (+/- {cv_auc.std():.3f})")
print(f"Recall: {cv_recall.mean():.3f} (+/- {cv_recall.std():.3f})")
# Expected output:
# AUC-ROC: 0.855 (+/- 0.025)
# Recall: 0.740 (+/- 0.040)
Success
Low standard deviation across folds means the model is stable. If std > 0.05 on AUC, the model is sensitive to which customers end up in the test fold — you may need more data, stronger regularisation, or simpler features.