Skip to content

Interview Questions — Customer Churn Prediction

These questions reflect what a data science interviewer will ask after you walk them through a churn prediction project. They test whether you understand the modelling decisions you made and whether you can connect those decisions to the business problem.

Q1. The churn rate is 30%. Is that a class imbalance problem? How did you handle it?

Show answer

30% is moderate imbalance — severe enough to make accuracy a misleading metric, but not so extreme that you need aggressive resampling techniques like SMOTE.

A classifier that always predicts "not churned" achieves 70% accuracy while catching zero churners. That is the baseline you need to beat. Reporting only accuracy here would give a false picture of model performance.

Handling approach in this project:

  1. class_weight='balanced' on logistic regression and random forest. This tells the algorithm to penalise misclassification of the minority class (churners) more heavily. Internally it is equivalent to upsampling churners by a factor of approximately 70/30 = 2.3x, but without actually duplicating rows.

  2. compute_sample_weight('balanced') on gradient boosting, which does not support a class_weight parameter natively. This achieves the same effect by passing observation-level weights to the fit call.

  3. Switched the primary metric from accuracy to recall on the churn class. The business cost of missing a churner (losing the customer) far exceeds the cost of a false positive (one unnecessary retention call). Optimising for recall aligns the model with the actual business objective.

  4. Lowered the decision threshold from 0.5 to 0.35, further increasing recall at the cost of some precision. This is appropriate given the asymmetric cost structure.

What I did NOT do: SMOTE or other synthetic oversampling. SMOTE can help when the imbalance ratio is 10:1 or worse. At 7:3 the class_weight approach is simpler, introduces no synthetic data artifacts, and works cleanly inside a Pipeline.

Q2. Why did you optimise for recall instead of accuracy?

Show answer

Because the costs of the two types of errors are not equal.

  • False negative (missing a churner): the customer cancels without any intervention. You lose their entire remaining lifetime value — months of future revenue, permanently.
  • False positive (calling a loyal customer): one unnecessary retention call. The cost is approximately 15 minutes of agent time.

When the cost of a false negative is 50–100x greater than the cost of a false positive, you optimise for recall — even at the expense of precision. Accuracy averages performance across both classes and treats every error as equally costly. It is the wrong metric for this cost structure.

The practical framing: a model with 88% recall and 55% precision catches 52 of 59 churners and generates 43 unnecessary calls. A model with 85% precision and 45% recall catches only 27 churners and wastes the other 32. The second model "sounds better" by precision but costs the business roughly 25 lost customers per cycle.

In an interview, always ground the metric choice in the cost of each error type before the interviewer asks you to justify it.

Q3. How did you decide on the decision threshold?

Show answer

The default threshold of 0.5 is a convention, not an optimum. I chose 0.35 through a combination of business constraint and data analysis.

Process:

  1. Plotted the precision-recall curve across all thresholds from 0.10 to 0.90.
  2. Computed F1 at every threshold and identified the threshold that maximised F1 (approximately 0.40–0.45 in this dataset).
  3. Cross-referenced with the business constraint: the retention team can handle roughly 150 calls per week. At threshold 0.35, the model flags ~95 customers in the 200-customer test set — scaling to ~475 per 1,000 — which is within team capacity for a mid-sized customer base.
  4. Built a threshold analysis table showing precision, recall, F1, and "customers flagged per 1,000" at five thresholds (0.30, 0.35, 0.40, 0.45, 0.50) and presented it to the stakeholder to choose the operating point.

Key principle: the threshold decision is partly statistical (F1-maximising point) and partly operational (how many calls can the team make). That second input can only come from the business. Show them the table; let them choose.

In production, the threshold should be revisited every time the team's capacity or the customer base changes. A static threshold from six months ago may no longer be appropriate.

Q4. What is calls_per_tenure and why did you create it instead of using raw features?

Show answer

calls_per_tenure = support_calls / (tenure_months + 1)

It captures contact intensity — the rate of support friction relative to how long the customer has been around.

The raw support_calls feature conflates two very different situations:

  • A customer who made 8 support calls in their first 2 months is in crisis. That is a rate of ~4 calls per month and a strong churn signal.
  • A customer who made 8 support calls over 5 years is a normal, engaged user. That is a rate of ~0.13 calls per month and is not a churn signal at all.

Both customers have support_calls = 8, but the first has a calls_per_tenure of ~3.7 and the second has ~0.13. The engineered feature separates what the raw count blurs together.

This is an example of ratio-based feature engineering: when two raw features interact to produce meaning, divide them. Other examples in the same spirit: revenue_per_user, cost_per_click, errors_per_request.

The + 1 in the denominator is a standard smoothing trick to prevent division by zero for any customer with tenure_months = 0.

Q5. A stakeholder says "83% accuracy sounds great!" How do you respond?

Show answer

Without dismissing their reaction, I would walk them through what 83% accuracy actually means in this context.

"83% accuracy sounds strong, but let me show you what that means in terms of customers. Our churn rate is 30%, so a model that predicts 'retained' for everyone — without looking at a single feature — gets 70% accuracy automatically. So 83% is only 13 percentage points better than doing nothing at all.

The number that actually matters to us is: of the 300 customers who will churn this month, how many did we catch? At 83% accuracy our model might only be catching 45–50 of them. That means 250+ customers are leaving while we do nothing, because the model we're calling 'great' missed them entirely.

The metric we should track is recall on churners — the percentage of actual churners we identified before they left. At threshold 0.35, our model catches 88% of churners. That is the number that drives retention revenue."

Then show the confusion matrix with the business translation. Most stakeholders understand "52 customers caught vs. 7 missed" immediately, even if precision and recall feel abstract.

Q6. How would you explain this model's output to the retention team who will use it?

Show answer

I would avoid showing them any probability scores, thresholds, or model metrics in the first conversation. Instead, I would frame it as a ranked priority list.

"Every Monday morning you will receive a list of customers sorted from highest to lowest churn risk. The top 150 on that list are your priority calls for the week. The model has identified these customers as showing signs they may cancel — things like a high volume of support calls relative to how long they've been with us, or not having a contract.

You don't need to know the technical details. What you do need to know: - The list is right about 55% of the time — roughly 1 in 2 calls will be to a genuinely at-risk customer. - When you call someone who turns out to be fine, treat it as a check-in. There is no harm in a proactive call to a happy customer. - Please log the outcome of every call: did the customer confirm they were considering leaving? Did they respond to the retention offer? This feedback helps us improve the model next month."

The two key messages for the team: (1) trust the ranked list, not their gut, for prioritisation; (2) log outcomes so the model can be retrained on real intervention data.

Q7. What additional data would most improve this model?

Show answer

Ranked by expected impact:

1. Product usage data (highest impact). Does the customer log in regularly? Have they used all the features they're paying for? Low engagement is the single strongest early churn signal in most SaaS datasets, and it is not captured in this dataset at all. A feature like days_since_last_login or feature_adoption_score would likely become the top predictor.

2. Support ticket sentiment and resolution time. The current model knows how many support calls a customer made, but not whether those calls were resolved satisfactorily. A customer who called five times and had all issues resolved is very different from one who called five times with no resolution. Unresolved tickets and negative CSAT scores are strong churn predictors.

3. Payment behaviour. Late payments, failed transactions, and downgrades are forward-looking churn signals. A customer who recently downgraded from Enterprise to Premium is signalling unhappiness before they cancel.

4. Competitor activity signals. In B2B markets, a customer whose domain shows up in competitor trial signups or who has been contacted by a competitor's sales rep is at elevated risk. This data is harder to get but extremely valuable.

5. Historical churn reason codes. If the retention team logs why customers left (price, product gaps, competitor, company shutdown), you can build separate models for each churn reason type and route at-risk customers to the right specialist.

Q8. How would you monitor this model in production to detect when it is degrading?

Show answer

Model monitoring has two separate problems: detecting when predictions are wrong, and detecting when they are going to become wrong.

Performance monitoring (lagged — you need ground truth). Once actual churn outcomes are known (30 days after prediction), compare the model's predictions against reality. Track AUC-ROC, recall on churners, and precision on churners on a rolling monthly basis. If AUC drops more than 5 percentage points from the deployment baseline, trigger a model review. A sudden drop usually means a data pipeline change or a real shift in customer behaviour.

Distribution monitoring (immediate — no ground truth needed). Track the distribution of input features and predicted probabilities weekly. Specifically: - Has the average calls_per_tenure shifted? (Could indicate a product issue or a support team change) - Has the mean predicted churn probability drifted? (If the model suddenly thinks 50% of customers are at risk when it used to say 30%, something changed upstream) - Are there new values appearing in segment or region that the encoder has not seen?

Use PSI (Population Stability Index) to quantify feature distribution shifts. PSI > 0.2 on any key feature is a strong signal that retraining is needed.

Business metric monitoring. Track the conversion rate of retention calls: what percentage of contacted at-risk customers actually responded to the retention offer and stayed? If this drops, the model may still be predicting churn correctly, but the intervention itself may no longer be effective — a different kind of problem.

Retraining cadence. In most subscription businesses, retrain monthly on the most recent 12 months of data. Use a time-based train-test split — never shuffle historical data for a time-series-adjacent problem, because leakage from future data is a real risk.

← Evaluation