Case Study Practice¶
Case study questions test something different from technical questions. They are not checking whether you know the definition of a word. They are checking whether you can take a vague business problem, structure it into a solvable data problem, make reasonable assumptions out loud, and arrive at a coherent recommendation.
The most common failure mode: candidates jump straight to model selection without clarifying what they are trying to solve. The second most common failure: they describe a technically correct approach but never connect it back to a business decision.
Use this framework for every case study.
The Framework: Four Steps¶
1. Clarify — ask 2–3 questions before saying anything about your approach. Interviewers expect this. Jumping straight to a solution looks like poor problem definition skills.
2. Structure — state your approach before diving in. "Here is how I would break this down." This gives the interviewer a map and signals organised thinking.
3. Analyse — work through the substantive steps. Be explicit about decisions and the reasoning behind them.
4. Recommend — close with a clear recommendation. "Given what we discussed, I would recommend X because Y. The main risk is Z, which I would monitor by tracking W."
Tip
When you are stuck, say what you are thinking, not just what you know. "I am trying to decide between precision and recall here — the choice depends on whether incorrectly flagging a good customer is worse than missing a churner." This is the reasoning interviewers want to hear.
Case Study 1: Metric Drop Investigation¶
Prompt¶
"Our weekly active users dropped 15% last Tuesday. Your manager just pinged you. What do you do?"
Framework Applied¶
Clarify: - Is this a confirmed drop in WAU or a dashboard/instrumentation issue? - Is the 15% calculated against last week, last month's average, or last year (same day)? - Did anything ship last Tuesday — a product change, a pricing change, an email campaign? - Is the drop global or limited to a segment (platform, geography, user type)?
Structure: Before drawing any conclusions, I need to rule out measurement error, then identify whether this is internal (something we did) or external (something that happened in the world).
Analyse:
Step 1 — verify the data: - Check the logging pipeline. Did any event tracking break? Are WAU numbers for prior periods unchanged? - Check if the drop is present in multiple data sources (analytics dashboard, database query, third-party tool).
Step 2 — segment the drop: - Break by platform (iOS, Android, web) - Break by user cohort (new vs. returning, age of account) - Break by geography - Break by feature usage
A drop that affects only one platform or one geography is much more informative than a uniform drop.
Step 3 — correlate with internal events: - Did a deployment go out on or before Tuesday? - Did an email campaign or push notification fire? - Did prices or subscription terms change?
Step 4 — correlate with external events: - Was there a competitor announcement? - Was there news coverage (negative or positive)? - Did a major third-party dependency (login provider, payment processor) have an outage?
Recommend:
"Based on this investigation, I would present a segmented breakdown within 30 minutes and a root cause hypothesis within 2 hours. If the drop is segment-specific and correlates with a recent deployment, I would escalate to the engineering team immediately. If it is measurement error, I would confirm and close the ticket. I would not speculate about root cause until the segmentation is done."
Warning
The most common mistake on this case: immediately jumping to "we should build a model to predict WAU drops." Metric investigation is first a data quality and segmentation problem, not a modelling problem. Show that you check your data before building anything.
Case Study 2: A/B Test Design¶
Prompt¶
"Our product team wants to test a new onboarding flow. They want your help designing the experiment."
Framework Applied¶
Clarify: - What is the primary metric we want to improve? Day-7 retention? Activation rate? Time to first action? - What is the current baseline rate for that metric? - What is the minimum detectable effect (MDE) we care about? - Are there any constraints — user groups we should not experiment on, regulatory regions, seasonality concerns? - How will the treatment be delivered — is it a UI change, a messaging change, a flow change?
Structure:
I will walk through: unit of randomisation, sample size calculation, duration, success/guardrail metrics, and analysis plan.
Analyse:
Unit of randomisation: randomise at the user level, not the session level. If a user sees both flows at different sessions, you contaminate the experiment.
Success metric: say the goal is Day-7 retention (proportion of users who are active 7 days after sign-up). Baseline: 35%.
Sample size: use a two-proportion z-test to determine required N per group.
from statsmodels.stats.power import NormalIndPower
analysis = NormalIndPower()
# Parameters
baseline = 0.35 # current Day-7 retention
mde = 0.02 # minimum lift we care about (2 percentage points)
alpha = 0.05 # significance level
power = 0.80 # 1 - Type II error rate
effect_size = (baseline + mde - baseline) / (baseline * (1 - baseline)) ** 0.5
n_per_group = analysis.solve_power(
effect_size=mde / (baseline * (1 - baseline)) ** 0.5,
alpha=alpha,
power=power,
alternative='two-sided'
)
print(f"Required sample size per group: {n_per_group:.0f}")
Duration: run for at least 2 full weeks to capture weekly seasonality. Do not stop early because the result looks significant — this inflates Type I error (peeking problem).
Guardrail metrics: monitor alongside the primary metric — things you do not want to move negatively: support ticket volume, account deletions, payment failure rate.
Analysis: at the end of the pre-specified duration, run a two-proportion z-test or chi-squared test. Do not run multiple comparisons without correction.
Recommend:
"I would run this for 2 weeks with a pre-registered success criterion of p < 0.05 on Day-7 retention with a minimum lift of 2 percentage points. I would check guardrail metrics at the midpoint, not to make a ship decision, but to catch catastrophic regressions. I would not look at the primary metric until the test is complete."
Warning
The most common mistake: "we will run it until we see significance." This is p-hacking. Early stopping inflates Type I error badly. Always pre-specify the duration and the stopping criteria before the test starts.
Case Study 3: Build a Churn Model From Scratch¶
Prompt¶
"A subscription company wants to reduce churn. Describe how you would build a model to predict which customers are likely to cancel next month."
Framework Applied¶
Clarify: - How is churn defined — cancellation, non-renewal, or inactivity for N days? - What is the churn rate currently? (Affects class imbalance strategy) - What data is available — CRM data, usage logs, billing history, support tickets? - What is the intervention — if we predict someone will churn, what do we do about it? (This affects which metric matters most: precision vs recall) - What is the cost of the intervention? (Offering a 20% discount to someone who was not going to churn is not free)
Structure:
I will cover: problem framing, data preparation, feature engineering, model selection and training, evaluation, and deployment considerations.
Analyse:
Problem framing: binary classification. Label = 1 if the customer cancelled in the next 30 days, 0 otherwise. The prediction window is "as of today" — you predict for customers who are currently active.
Data preparation:
import pandas as pd
from sklearn.model_selection import train_test_split
# Define the label
# For each customer, at the snapshot date, did they churn in the next 30 days?
df['churned_next_30'] = (df['days_to_churn'] <= 30).astype(int)
# Temporal split: train on data before a cutoff, validate on data after
cutoff = '2023-06-01'
train = df[df['snapshot_date'] < cutoff]
val = df[df['snapshot_date'] >= cutoff]
X_train = train.drop(columns=['churned_next_30', 'customer_id', 'snapshot_date'])
y_train = train['churned_next_30']
X_val = val.drop(columns=['churned_next_30', 'customer_id', 'snapshot_date'])
y_val = val['churned_next_30']
Feature engineering — what to include:
- Recency: days since last login, last purchase
- Frequency: logins per week over past 30/60/90 days
- Billing: payment failures, plan tier, months since upgrade/downgrade
- Engagement: feature usage counts, support tickets opened
- Trend: is engagement going up or down? (e.g., logins_last_30 / logins_prev_30)
Model selection:
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, classification_report
# Start with logistic regression as the baseline
baseline = LogisticRegression(class_weight='balanced', max_iter=1000)
baseline.fit(X_train, y_train)
baseline_auc = roc_auc_score(y_val, baseline.predict_proba(X_val)[:, 1])
print(f"Baseline AUC: {baseline_auc:.3f}")
# Then try gradient boosting
gbm = GradientBoostingClassifier(n_estimators=200, max_depth=4, learning_rate=0.05)
gbm.fit(X_train, y_train)
gbm_auc = roc_auc_score(y_val, gbm.predict_proba(X_val)[:, 1])
print(f"GBM AUC: {gbm_auc:.3f}")
Evaluation:
The right metric depends on the intervention design: - If the team calls every predicted churner, recall matters more — you want to catch as many real churners as possible. - If the intervention is expensive (large discount), precision matters — you do not want to waste budget on non-churners. - AUC-ROC gives a threshold-agnostic view of discriminative ability.
Deployment considerations: - Predictions should be refreshed on a schedule aligned with the intervention cadence (weekly, daily) - Monitor for data drift: if the distribution of input features shifts, model performance will degrade silently - Track actual churn rates in the "predicted to churn" bucket and the "not predicted to churn" bucket to validate lift
Recommend:
"I would deliver a gradient boosting model targeting AUC > 0.80 on held-out data, with precision and recall tuned at the threshold that maximises the retention team's intervention budget efficiency. I would set up a monitoring dashboard tracking AUC and calibration monthly, with a retrain trigger if AUC drops below 0.75."
Warning
The most common mistake: using a random 80/20 split for churn modelling. Churn data is temporal. Randomly splitting means some validation examples are from before training examples, which is leakage. Always split by time.
Case Study 4: Product Analytics Question¶
Prompt¶
"We have two groups of users: those who use Feature X and those who do not. Users who use Feature X have 40% higher 30-day retention. Should we push Feature X to all users?"
Framework Applied¶
Clarify: - How is "use Feature X" defined — any interaction, N interactions, a specific depth of engagement? - Is this from an experiment or an observational analysis? - What is the time period of this analysis? Was it computed on a representative user cohort?
Structure:
This is fundamentally a causality question, not a correlation question. The 40% lift tells us that feature X users retain better — but it does not tell us whether feature X caused the retention or whether the users who seek out feature X are already more engaged.
Analyse:
The selection bias problem:
Users who discover and use Feature X are likely not a random sample of your user base. They may be your most engaged, most motivated users — people who would have high retention regardless of feature X. If you push Feature X to all users (including less engaged ones), you may see much smaller or no retention lift.
This is called survivorship bias (or more precisely, selection on the dependent variable).
How to answer the causal question:
Option 1 — Run an A/B test. Randomly assign new users to a group that receives an aggressive Feature X promotion vs. a control group. Measure 30-day retention. This is the gold standard.
Option 2 — Instrumental variables or propensity score matching. If an experiment is not possible, use observational causal inference methods to create a more comparable comparison group.
Option 3 — Look for natural experiments. Did Feature X roll out to some users before others due to a technical reason? If so, compare retention for users who got access early vs. late — this approximates a natural experiment.
What the 40% lift actually tells you:
# Illustrative: check if "feature X users" differ systematically from non-users
feature_x_users = df[df['used_feature_x'] == 1]
non_feature_x_users = df[df['used_feature_x'] == 0]
print(feature_x_users[['age_days', 'sessions_week1', 'plan_tier']].mean())
print(non_feature_x_users[['age_days', 'sessions_week1', 'plan_tier']].mean())
# If feature X users have 3x the Week 1 sessions, the retention gap is likely driven
# by engagement level, not by Feature X itself
Recommend:
"I would not push Feature X to all users based on this observational analysis alone. The 40% retention lift may reflect selection bias — Feature X users are likely already your most engaged cohort. Before any rollout, I would run a 4-week A/B experiment where we actively promote Feature X to a random subset of new users and measure 30-day retention versus control. If we see a statistically significant lift of at least 10 percentage points in the experiment, then we have evidence to justify the rollout."
Success
Calling out selection bias and asking for a controlled experiment is the correct answer here. Many candidates see the 40% number and immediately say "yes, roll it out." The stronger candidate recognises the correlation vs. causation trap and proposes the right analytical test.
Warning
The most common mistake: accepting the 40% lift at face value and recommending a full rollout without questioning whether this is a causal relationship. This is one of the most common analytical errors in product data science.