Business Case Walkthrough¶
End-to-end case walkthroughs are the most demanding question type in data science interviews. In 20–30 minutes, you are expected to demonstrate that you can take a vague business problem all the way through to a deployed, monitored system — making defensible decisions at each step. This file walks through four complete cases with structured frameworks you can adapt.
How to Structure a 30-Minute Business Case Answer¶
Before the walkthroughs, understand the skeleton every answer should follow:
- Clarify the problem (2–3 minutes): ask 2–3 targeted questions to pin down scope, business context, and what "success" means
- Define metrics (2–3 minutes): primary metric, guardrail metrics, and how you will evaluate the ML model
- Data inventory (2–3 minutes): what data do you have, what data do you need, what are the label sources
- Model design (5–7 minutes): feature engineering, model family, why this choice for this problem
- Evaluation (3–4 minutes): offline evaluation, how you validate before deployment
- Deployment and rollout (2–3 minutes): how the model integrates with the product, A/B test design
- Monitoring (2–3 minutes): what signals tell you the model is working or failing post-launch
What makes the difference between a pass and a strong pass: raising edge cases and tradeoffs at each step without being prompted. The interviewer should be nodding, not guiding.
Case 1: Spotify's Podcast Recommendation Is Underperforming — Diagnose and Fix¶
The prompt: Podcast listening has been growing, but our podcast recommendation algorithm is generating much lower engagement than music recommendations. Diagnose what's wrong and propose a fix.
Part A: Diagnostic — what could cause low engagement?¶
Show answer
Step 1: Clarify the numbers Before diagnosing, establish what "much lower" means: - Clicks per recommendation impression? (Low CTR suggests bad surface-level relevance) - Completion rate? (Low completion means users start episodes and abandon — bad match on content, length, or quality) - Return rate to podcasts? (If users rarely return to podcast recommendations after a first click, that is a retention problem, not a discovery problem) - How does engagement compare across different recommendation surfaces (homepage vs end-of-episode autoplay vs search)?
Step 2: Hypotheses by layer
Data quality issues: - Podcast metadata is sparse compared to music. Songs have structured genres, moods, BPM, key, lyric-derived features. Podcasts often have only a title, description, and publisher. - Episode-level vs show-level signals: a user who loved Episode 3 of a podcast should be recommended Episode 4 — but if the system treats each episode as an independent item, this continuity signal is lost. - Cold start dominates: new podcast episodes have no engagement data, unlike established music tracks. A model that relies heavily on collaborative filtering will chronically underrank new episodes.
Label definition problems: - Is the model trained to predict clicks or listen-completions? Clicking a 90-minute episode and listening for 5 minutes is not success. Completion rate is a better label but arrives with a delay. - Podcast consumption is episodic — a user might binge a show for a week, finish it, and not return. The model may interpret this as churn when it is actually normal behaviour.
User intent differences: - Podcast listening is often intentional: users search for a specific topic or show. Recommendations work differently when users have a clear intent vs when they want ambient discovery. - Podcast sessions are longer and more deliberate. A wrong recommendation wastes 30+ minutes of a user's time, more than a bad song recommendation. The threshold for "good enough" is higher.
Step 3: Investigation plan - Disaggregate engagement by episode age (new vs established episodes) — if new episodes are chronically underranked, cold start is the issue - Disaggregate by show type (narrative fiction, news, interview, education) — if one category has especially poor engagement, content-based features for that category may be missing - Look at the features driving recommendation rankings today — if the top features are all collaborative filtering signals, cold start explains the underperformance - Run qualitative analysis: pull 50 examples of low-engagement recommendations and manually assess — are they obviously bad matches, or are they plausible but badly timed?
Part B: Proposed fix — model redesign¶
Show answer
Feature engineering improvements: - Use audio/transcript embeddings from the episode content itself to enable content-based similarity when engagement data is sparse - Add show-level features in addition to episode-level: overall show listen history for this user, show genre, publisher reputation score - Add temporal context: listening at 6 AM likely signals a commute (news, short content preferred); 10 PM signals leisure (longer narrative content preferred) - Explicitly model the "continuing listener" signal: if a user has listened to Episodes 1 and 2 of a show, Episode 3 should rank very high for them regardless of collaborative filtering signals
Model architecture: - Two-stage retrieval + ranking: use a fast embedding model to retrieve 500 candidates from the full catalogue, then a more expensive ranker to order the top 50 for display - Hybrid model: combine collaborative filtering (for established episodes with engagement history) with content-based signals (for new episodes with no history)
Label redefinition: - Primary training signal: 50%+ episode completion (not just click) - Secondary: return engagement (did the user listen to another episode from the same show within 7 days?) - Negative signals: < 5 minute listen before abandonment, explicit skip
Evaluation: - Offline: NDCG@10 on a held-out sample, disaggregated by episode age bracket (0–7 days, 7–30 days, 30+ days) - Online: A/B test against existing recommender, primary metric = 50%+ completion rate per recommendation impression, guardrail = overall podcast listening minutes per user (don't cannibalise total consumption)
Monitoring post-launch: - Weekly: check completion rate by episode age to confirm cold start improvement - Monthly: check show-level diversity in recommendations (are users receiving variety across show types, or are they being locked into one show?) - Data drift: podcast catalogue growth is rapid. Monitor feature distributions for new episodes monthly.
Case 2: Design a System to Detect Fraudulent Ride-Sharing Accounts¶
The prompt: Fraudsters are creating fake accounts on our ride-sharing platform and using stolen payment information. Design an ML system to detect and block them.
Part A: Problem framing¶
Show answer
Clarifying questions: - What types of fraud are in scope? Account takeover (legitimate account credentials stolen), synthetic identity (fake accounts created at sign-up), or payment fraud (stolen cards used in real accounts)? - What does "block" mean operationally? Automatic account suspension, flagging for human review, increasing friction (require additional verification)? - What is the tolerance for false positives? Blocking a legitimate rider is high friction — they will contact support, may churn, may leave a negative review. - What is the current baseline? Is there an existing rule-based fraud system? What is the current fraud rate?
Defining the prediction task: - Unit of prediction: the account (at creation time, at transaction time, or both?) - At account creation: predict whether a newly created account is synthetic/fraudulent - At transaction time: predict whether a specific ride booking or payment is fraudulent - A two-layer system — account-level risk score + transaction-level risk score — is more robust than either alone
Defining success: - Primary metric: fraud prevented per dollar of fraud attempted (recall at a specified precision threshold) - Business framing: if the fraud rate is 0.5% and we can reduce it to 0.1%, what is the dollar value of that reduction? - Guardrail: false positive rate — the maximum acceptable fraction of legitimate accounts incorrectly blocked. Often expressed as "no more than X% of legitimate users should experience any additional friction"
Part B: Data and features¶
Show answer
Label sources: - Confirmed fraud labels: chargebacks, reports from payment processors, manual review outcomes, accounts banned via rule-based system - These labels are delayed (chargebacks arrive weeks after the transaction), sparse (0.5% fraud rate), and noisy (some fraud goes undetected, some legitimate chargebacks are mislabelled as fraud)
Feature categories:
Device and environment signals (available at sign-up and at transaction time) - Device fingerprint: is this device associated with many accounts? (Device multiplexing is a fraud signal) - IP address: datacenter IP? Tor exit node? IP seen in previous fraud? - Location consistency: does the GPS, IP, and phone area code align? Inconsistency is a signal. - App version and OS: fraudsters often use emulators or old app versions
Account behaviour signals (available post-signup) - Time from account creation to first ride: fraudsters often move fast - Number of accounts from same device, email domain, phone prefix - Payment method characteristics: prepaid card, new card added immediately before first ride - Ride patterns: routes that don't match the account's registration location
Network signals - Does this account share characteristics (device, phone number prefix, email pattern, IP block) with known fraud accounts? - Graph-based features: connections between accounts via shared devices or payment methods
Model choice: - Gradient boosting (XGBoost, LightGBM) for tabular features — strong performance on imbalanced structured data, handles missing values natively - Graph neural network or embedding layer for network features — captures relationships between accounts that tabular models miss - Two-stage: GBM for fast initial scoring (sub-10ms), optional GNN layer for accounts flagged as borderline
Part C: Evaluation and deployment¶
Show answer
Offline evaluation: - Use a time-based split: train on January–September, evaluate on October–December. Never shuffle and split randomly — fraud patterns evolve and random splits leak future information into training. - Primary metric: precision at fixed recall thresholds (e.g., precision at 80% recall). What fraction of flagged accounts are truly fraudulent, given that we are catching 80% of actual fraud? - Disaggregate by fraud type if labels allow
Deployment architecture: - Real-time scoring at account creation and transaction time (< 50ms latency budget) - Output: a risk score (0–1); threshold determines action (auto-block, flag for review, allow) - Maintain a review queue for high-score accounts that haven't been auto-blocked — human reviewers resolve borderline cases and feed confirmed labels back into the training pipeline
The exploration-exploitation tradeoff in fraud: - If you block every high-risk account, fraudsters quickly learn which behaviours trigger your system and adapt - A small fraction of high-risk accounts should be allowed through (with close monitoring) to observe how fraudsters behave post-signup — these are training examples for the next model version - This is called "allowing some fraud through for learning purposes" — a deliberate strategic choice, not an oversight
Monitoring post-launch: - Fraud rate (ground truth arrives delayed via chargebacks — build a 30-day lagging metric dashboard) - False positive rate: track support tickets from blocked users; distinguish "fraudster complaining" from "legitimate user blocked" - Data drift: fraudsters adapt faster than legitimate users. Monitor the distribution of device fingerprints, email domains, and IP ranges monthly. - Model decay: expect the model to lose effectiveness within 3–6 months as fraud patterns shift. Establish a retraining cadence.
Case 3: Our E-Commerce Search Is Not Converting Well — What Would You Do?¶
The prompt: Our site search converts at 4%, while users who browse categories convert at 7%. We believe search has a relevance problem. How would you investigate and fix it?
Part A: Investigation¶
Show answer
Clarify the numbers before assuming a solution: - How is conversion defined? Add-to-cart? Completed purchase? (These can diverge significantly) - Is 4% vs 7% for the same user types? (Search users and browse users may have different intent — searchers often have more specific intent but also higher abandonment if they don't find exactly what they want) - What is the search query distribution? Head queries (common, high-volume) vs tail queries (rare, specific)? Relevance problems often concentrate in the tail. - Is the gap consistent across product categories, or concentrated in one area?
Diagnostic approach:
Query analysis: - What fraction of searches return zero results? A high zero-results rate (> 5–10%) is a direct relevance failure — the index doesn't contain what users are asking for. - What fraction of searches result in an immediate reformulation? If 30% of users rephrase their query immediately, the first results were bad. - What is the click-through rate on position 1 vs position 5? A flat distribution suggests the ranking is not well-calibrated.
Session analysis: - Do users who search and don't convert then go to browse categories and convert? If yes, search is failing to surface what users already want. - What is the search-to-purchase funnel? Search → click → add-to-cart → checkout. Where does it drop?
Qualitative review: - Pull 100 representative queries from the last 30 days. For each, manually assess: are the top 5 results relevant? If a human curator would have different results, the model has a ranking problem.
Part B: Model redesign¶
Show answer
Feature engineering for search ranking:
- Query-item relevance features: BM25 score (traditional term-matching), semantic similarity (query and product description embeddings), exact match signals (query term in product title vs description vs reviews)
- Item quality signals: average rating, number of reviews, inventory level, return rate
- Behavioural signals: click-through rate on this item for this query (or similar queries), add-to-cart rate, conversion rate
- User personalisation: does the user's browse/purchase history indicate a preference for specific brands, price ranges, or product attributes relevant to this query?
Model architecture: - Two-stage retrieval + ranking: BM25 or ANN retrieval for initial candidate set (10,000 items → 500); learned ranker (LambdaMART, LightGBM with ranking objective) for final ordering - Use LTR (Learning to Rank): train on implicit labels (clicks), but weight by downstream conversion signal
Addressing zero-result queries: - Spelling correction and query expansion using product catalogue vocabulary - Query understanding: if a user searches "running shoes for flat feet," extract the intent (running shoes) and the constraint (flat feet) and map to filterable attributes
Evaluation: - Offline: NDCG@5 and MRR (Mean Reciprocal Rank) on a manually labelled query set - Online: A/B test with primary metric = conversion rate per search session; guardrail = search refinement rate (don't increase the fraction of users who reformulate — that would indicate the new model is worse)
The key insight in this case: search is an information retrieval problem where the user's intent is partially observable from the query and partially inferred from context. A good answer acknowledges this ambiguity and designs features accordingly — not just matching keywords.
Case 4: Reduce Churn for a SaaS Product Using ML — Where Do You Start?¶
The prompt: We have a B2B SaaS product with 5,000 enterprise customers. Monthly churn is running at 2.5%, which is too high. We want to use ML to predict churn and intervene. Where do you start?
Part A: Problem framing¶
Show answer
Clarifying questions: - What is the unit of churn? The enterprise account (company level) or individual user seats within an account? (B2B churn is almost always at the account level, but leading signals often come from user-level behaviour) - What does "intervene" mean? Automated outreach? CS team notification? Discount offer? The intervention determines how early you need the prediction and what precision you need. - What is the contract structure? Month-to-month vs annual? Annual contracts have a single renewal decision point; month-to-month churn can happen any time. - What data do you currently have on product usage?
Defining the prediction target: - Binary classification: will this account churn in the next 90 days? (90 days is chosen because the CS team needs enough lead time to intervene) - Training label: "churned" = account cancelled or failed to renew. This label arrives at renewal, which may be months away.
What makes B2B churn different from B2C: - Fewer accounts (5,000 vs millions for B2C): the dataset is small; a simple model trained on 5,000 accounts will be noisy. Cross-validation is essential. - Multiple users per account: the account decision is made by a buyer/manager, but the signal comes from end user behaviour. You need to aggregate user-level signals to account level. - Sales and CS relationship data: sentiment from CS notes, number of support tickets, engagement with CSM. This is qualitative but highly predictive.
Part B: Data and features¶
Show answer
Feature categories for B2B churn:
Product usage (highest signal) - Daily/weekly active users per account as a fraction of licensed seats (seat utilisation) - Time since last login across all users in the account - Number of core features used in the last 30 days (breadth of adoption) - Integration activity: are they using API integrations? (Deep integrations correlate with lower churn — switching cost is higher) - Usage trend: is seat utilisation increasing, flat, or declining over the last 60 days? (Trend often predicts churn before the absolute level does)
Customer health and relationship signals - Number of support tickets in last 30 days and their severity/resolution time - NPS score from last survey - CSM engagement: last CSM touchpoint date, last executive business review date - Renewal stage: is the account in renewal conversation? Is there a contract expiry in the next 90 days?
Company context (external data) - Account ARR (higher ARR accounts churn less — they have more to lose) - Account tenure (longer tenured accounts churn less) - Company firmographics: industry, headcount, growth stage (can be enriched from LinkedIn, Clearbit)
Model choice: - With 5,000 accounts, tree-based models (XGBoost, LightGBM) are well-suited — they handle mixed feature types, are robust to noise, and don't require large datasets - Logistic regression as a baseline and for interpretability (CS team needs to understand why an account is flagged) - SHAP values for account-level explanations: "this account is flagged because seat utilisation dropped 40% in the last 30 days and they haven't logged a support ticket in 60 days (suggesting low engagement, not active problem)"
The label challenge: - With 5,000 accounts and 2.5% monthly churn, you have ~125 churn events per month. At annual cadence, ~1,500 churns total in the training history. - This is a small dataset — use leave-one-out or stratified 5-fold CV, not a simple train/test split. Every labelled example matters.
Part C: Intervention design and evaluation¶
Show answer
Intervention design: The model output should feed an intervention tier system based on predicted churn probability and account ARR: - High probability + high ARR: immediate CSM escalation, executive outreach, discount authority - High probability + low ARR: automated email sequence, self-serve resources, in-app health score nudge - Medium probability: add to CSM watchlist, schedule proactive check-in
Evaluation challenges specific to this problem:
Causal attribution: if a CSM reaches out to a high-risk account and they don't churn, how do you know whether the intervention caused them to stay or whether the model was wrong (they weren't actually going to churn)? - This is the counterfactual problem. A holdout design — where a small fraction of predicted high-risk accounts receive no intervention — is the only rigorous way to measure intervention effectiveness. - Practical challenge: withholding intervention from high-risk accounts is ethically and commercially uncomfortable. Frame it as "testing which type of intervention works best" rather than "some accounts get no help."
Monitoring and retraining: - B2B churn is often driven by market events (competitor launches, economic downturns, leadership changes at customer companies) that happen faster than monthly retraining cadences - Build a "human-in-the-loop" override: CS managers should be able to flag accounts the model missed or incorrectly scored. These overrides feed labelled training data for the next model version.
Communicating to the business: - Report churn risk score by segment, ARR band, and product line — not just an aggregate number - Track the health score distribution over time: is the overall population of accounts getting healthier or sicker? - The model is a tool for the CS team, not a replacement. The answer to "will this account churn?" is never a certainty — frame outputs as risk scores and let the CS team exercise judgment.
Common Interview Patterns and How to Prepare¶
Show answer
Pattern 1: The metric diagnostic "X is down. What do you do?" — always: verify data, then segment (platform/geo/cohort/funnel step), then check internal changes, then check external causes. Never skip step 1 (data quality).
Pattern 2: The product design case "How would you build/improve/launch X?" — structure: clarify scope → define metrics → data and labels → model design → evaluation → deployment → monitoring. Don't skip monitoring; it signals you've shipped models before.
Pattern 3: The estimation/back-of-envelope "How many ride requests does Uber get per day?" — show your reasoning step by step, sanity-check your number at the end. Interviewers care about the approach, not whether you land on the exact number.
Pattern 4: The tradeoff question "Should we optimise for precision or recall?" — always answer "it depends on the cost of each error type, and that is a business question." Name the stakeholder who should make that call.
What separates strong candidates: - They ask clarifying questions before diving in (you cannot solve the right problem if you don't know what it is) - They are explicit about assumptions ("I'll assume X for now — is that reasonable?") - They raise edge cases without being prompted ("one thing I want to flag is the cold start problem for new items...") - They connect every technical decision to a business rationale ("I chose recall over precision here because the cost of missing a fraud case is 100× the cost of a false positive") - They know what they don't know ("I'd want to validate this assumption with the data team before committing to this approach")
← Previous: ML Problem Framing | → Next: Behavioral — Telling Your Story