ML Problem Framing¶

Problem framing questions separate data scientists who can build models from those who can build the right model. Interviewers use these questions to assess whether you understand the full lifecycle — from business problem to deployed system — not just the modelling step in the middle.

Q1: A product manager says "we want to use ML to improve our app." How do you turn this into an actionable ML task?¶

Show answer

This kind of vague ask is the most common starting point for real ML projects. Your job is to translate a business intent into a concrete ML formulation.

Step 1: Clarify the business problem Before touching any data or model, ask: - What is the specific outcome the business wants to change? (Increase revenue? Reduce churn? Improve engagement?) - What decision does the model need to support? Who makes that decision — a human or the system automatically? - What does success look like in 6 months? In numbers.

Step 2: Define the prediction target (output) - What exactly should the model predict? - Is it a classification task (user will churn: yes/no), regression (predicted LTV), ranking (order search results), or something else? - Can the output be directly measured in historical data?

Step 3: Identify training labels - Do you have ground truth labels in historical data? - How noisy are the labels? (User reported churn vs system-inferred churn) - Are labels available quickly enough to create a useful feedback loop?

Step 4: Define the features (input) - What data about the user/item/context is available at prediction time? - Is there anything that looks predictive but would not be available at inference time? (Data leakage risk)

Step 5: Define the system and deployment context - When does the prediction need to be made? (Real-time during a session vs batch overnight?) - What is the latency budget? - How does the model output connect to the product action?

Step 6: Define the success metric - What ML metric (AUC, RMSE, precision@K) aligns with the business goal? - What is the baseline to beat? (Current rule-based system, random, most-popular)

What separates good from great in an interview: not jumping to model selection. Most interviewers are testing whether you know to ask "what problem are we solving and what are we predicting?" before "what algorithm should we use?"

Q2: How do you choose between formulating a problem as classification vs regression vs ranking?¶

Show answer

The right formulation depends on what the downstream action is — not on what feels natural or what algorithm you know best.

Classification — when you need a discrete category or a yes/no decision - Predict whether a user will churn in the next 30 days - Classify a transaction as fraudulent or legitimate - Categorise a support ticket by issue type - Output: a probability + a decision threshold

Regression — when you need a continuous value - Predict the LTV of a new user - Estimate the demand for a product in a region - Forecast revenue for next quarter - Output: a continuous number; the model is evaluated on error magnitude

Ranking — when you need to order a set of candidates by relevance or predicted value - Rank search results by relevance to a query - Order product recommendations by predicted engagement probability - Prioritise outreach leads by predicted conversion likelihood - Output: a ranked list; the model is evaluated on ranking quality (NDCG, MRR, MAP)

How the same problem can be framed multiple ways: "Predict which users to send a promotional email to" can be: - Classification: will this user convert if emailed? (yes/no, calibrated probability) - Regression: what is the expected revenue lift from emailing this user? - Ranking: rank all users by expected conversion value and take the top N

Ranking is usually better when you have a fixed budget (top N emails, top K recommendations). Classification is better when the decision threshold matters (everything above X% gets an email). Regression is better when you need to optimise expected value.

The key question: what does the system do with the model output? That determines the right formulation.

Q3: How do you obtain training labels for an ML problem when ground truth is noisy or delayed?¶

Show answer

Labelling is the hardest, most under-discussed part of real ML problems. Good label design is often more valuable than good model architecture.

Common labelling challenges:

Noisy labels - User ratings (1–5 stars) are subjective and inconsistent; the same experience gets different ratings from different users - Explicit feedback (thumbs up/down) is sparse — most users never rate anything - Implicit feedback (clicks, plays, purchases) is denser but noisier — a click does not always mean satisfaction

Delayed labels - Fraud detection: it may take weeks to confirm a transaction is fraudulent (disputes, investigations) - Churn: you only know a user churned after the churn period has elapsed - Conversion: a user may convert weeks after first exposure

Sparse labels - Medical diagnosis datasets often have thousands of negatives per positive - Click data has millions of impressions with tiny fractions resulting in purchase

Strategies for handling label challenges:

1. Weak supervision Use heuristics, rules, or cheaper proxy labels to label data at scale. Example: use "user abandoned the page in < 5 seconds" as a weak negative label. Use Snorkel-style label functions and combine them with learned weights.

2. Human annotation For high-stakes or high-ambiguity tasks, invest in human annotation with clear guidelines and inter-annotator agreement measurement (Cohen's kappa). Sample-review labelled data before training.

3. Label smoothing If labels are known to be noisy, label smoothing regularises the model to be less confident about individual labels. Can also use confidence-weighted training.

4. Proxy labels If the true label arrives too late, find a correlated proxy that arrives faster. Churn prediction example: use "days since last session exceeds 7" as a proxy for churn risk.

5. Semi-supervised learning Label a small dataset; train on it; use the model's predictions on unlabelled data as soft labels for retraining. Requires careful validation that the semi-supervised labels are not introducing bias.

The interview answer quality signal: acknowledging that label quality is often the binding constraint in real projects, not model choice or compute.

Q4: How do you define the prediction target for an ML model without falling into proxy metric traps?¶

Show answer

A proxy metric trap occurs when the thing you optimise in training diverges from the business outcome you actually care about. It is one of the most common sources of deployed ML systems that "work technically" but fail commercially.

Classic proxy metric traps:

Watch time vs satisfaction (YouTube): optimising for watch time rewarded clickbait, misleading thumbnails, and addictive but low-quality content. Users reported watching things they regretted. YouTube added a satisfaction signal after recognising this divergence.
Clicks vs purchases (e-commerce): a product recommendation model trained to maximise clicks will surface curiosity-inducing but low-intent items. The right target is add-to-cart rate or purchase rate.
Open rate vs response rate (email): optimising for email open rate favours clickbait subject lines; optimising for reply rate or conversion favours relevance and personalisation.
Number of predictions vs useful predictions (fraud): a fraud model optimised for recall flags every transaction — maximises "caught fraud" but creates so much friction that users leave.

How to choose the right prediction target:

Start from the business outcome: what is the metric the business cares about? (Revenue, retention, satisfaction)
Trace the causal chain: how does the model output connect to the business outcome? Are there intermediate steps where the connection could break?
Validate alignment empirically: if your proxy metric and the true outcome are both observable in historical data, check their correlation. A proxy with <0.3 correlation to the true outcome is dangerous.
Define counter-metrics for your prediction target: what should not improve when you optimise for this target? Monitor that counter-metric.
Test offline before optimising: before committing to a training objective, check whether high scores on that objective in a held-out set correspond to high scores on the business metric.

Q5: What features can you use at inference time? Why does this constraint matter so much?¶

Show answer

Feature availability at inference time is a hard constraint that is often ignored during exploration and causes failures in deployment. A feature that is highly predictive in offline evaluation is worthless if it cannot be computed when the model needs to make a prediction.

Why this constraint is tricky: - In training, you work with historical data where all features exist simultaneously - In production, the model is called at a specific moment in time, and only features available at that moment can be used - The temptation is to use "future" features in training because they are highly predictive — but they would not exist at inference time

Common violation patterns:

Post-event features: using "number of purchases in the week after signup" to predict whether a user will become high-value. At signup time, this data doesn't exist.
Aggregations that include the target event: building a feature "total number of returns by this user" for a return prediction model — but the current return is included in the count.
Real-time data that is unavailable: a pricing model trained on live competitor prices, but the inference system cannot call the competitor API in real time.
External data with refresh delays: weather data that is refreshed daily, used in a model that makes predictions every hour.

The correct process: 1. Identify the "prediction moment" — exactly when in the system's lifecycle the model makes a prediction 2. For each candidate feature, ask: does this data exist at the prediction moment, and can it be computed within the latency budget? 3. For batch models (overnight scoring): features from yesterday's data are typically fine; features from today's activity may not be complete 4. For real-time models: only features that can be computed from a snapshot of existing data within milliseconds are valid

How to document feature availability: maintain a feature registry that records not just the feature definition but the data source, refresh frequency, and expected latency. This is standard practice in mature ML platforms.

Q6: How do you approach a problem with severe class imbalance — for example, fraud detection where 0.1% of transactions are fraudulent?¶

Show answer

Class imbalance is the norm, not the exception, in real-world ML problems. The approach depends on the severity of imbalance and the business cost of each error type.

Step 1: Understand the cost asymmetry Before any modelling, answer: what is the cost of a false positive (flagging a legitimate transaction as fraud) vs a false negative (missing a real fraud)? - In fraud detection: false negatives cost money (undetected fraud losses); false positives cost user friction (blocked legitimate users) - In cancer screening: false negatives cost lives; false positives cost unnecessary follow-up procedures - The cost ratio should inform your threshold setting and model evaluation metric

Step 2: Choose the right evaluation metric - Accuracy is useless: a model that predicts "not fraud" on every transaction gets 99.9% accuracy but catches 0% of fraud - Use precision, recall, F1, PR-AUC, or Matthews Correlation Coefficient - PR-AUC is particularly useful for severe imbalance — it focuses on the minority class

Step 3: Techniques at the data level - Undersampling the majority class: randomly remove majority samples. Fast, but loses information. - Oversampling the minority class with SMOTE: generate synthetic minority samples by interpolating between existing minority samples. Better than random duplication. - Combining both: undersample majority + oversample minority to reach a target ratio.

Step 4: Techniques at the algorithm level - Class weights: pass class_weight='balanced' to scikit-learn models — equivalent to oversampling by reweighting the loss function. - Threshold tuning: the default 0.5 threshold is not optimal for imbalanced problems. Choose the threshold that maximises the metric that matches your business cost structure (F1, or a custom cost-weighted metric).

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report, average_precision_score

model = RandomForestClassifier(class_weight='balanced', n_estimators=100, random_state=42)
model.fit(X_train, y_train)

# Use probabilities for threshold tuning
y_proba = model.predict_proba(X_val)[:, 1]

# PR-AUC is the right metric for severe imbalance
pr_auc = average_precision_score(y_val, y_proba)
print(f"PR-AUC: {pr_auc:.4f}")

What separates good from great: discussing threshold selection as a business decision, not a statistical one. The "best" threshold depends on the cost ratio of each error type — and that is a conversation with the product or business team, not something you derive from the data alone.

Q7: How do you define "success" for an ML model before training it?¶

Show answer

Defining success before training is one of the most important disciplines in applied ML. Without it, you have no objective way to know when to stop iterating or whether the model is ready to deploy.

The success definition should answer three questions:

1. What ML metric constitutes "good enough"? - Pick one primary offline metric aligned with the business objective (PR-AUC for fraud, NDCG for ranking, RMSE for regression) - Establish a baseline: what does a simple rule-based system or the existing model achieve? Your model should clearly beat this. - Set a numerical threshold: "the model must achieve PR-AUC > 0.85 on the holdout set" — not "the model should be as good as possible"

2. What is the minimum business impact required to justify deployment? - Online evaluation matters as much as offline: "the model must show a statistically significant lift in the primary product metric in an A/B test" - Define the minimum detectable effect that would justify the model's infrastructure cost and maintenance overhead

3. What are the non-negotiable constraints? - Latency: "the model must return a prediction in < 50ms for 99% of requests" - Fairness: "the model must not show more than X% disparity in false positive rate across demographic groups" - Explainability: "the model must produce a human-readable explanation for each decision" (required for regulated industries)

The accountability structure: Write the success definition in a model card or project brief before any training run. When the project reaches review, the discussion is about whether the pre-defined bar was met — not about whether the results look impressive. This prevents moving the goalposts after seeing results.

Q8: When should you NOT use ML? What are the signs that a simpler approach is better?¶

Show answer

Every problem that can be solved with a rule or a lookup table should be. ML adds complexity, maintenance overhead, and opacity. The bar for using ML should be: it clearly outperforms simpler approaches on outcomes that matter.

Signs that ML is not the right tool:

The data does not exist yet ML requires historical data with signal. If you are building a new product with no user data, there is nothing to train on. Start with rules or heuristics, collect data, then revisit ML after 3–6 months.

The relationship is deterministic or rule-based by design If the decision is "charge users a $10 cancellation fee if they cancel within 24 hours," that is a business rule — there is nothing to learn. ML cannot improve on a hard business policy.

There is no feedback loop ML systems improve because model outputs affect the world and the results are observed, creating training data. If your system makes predictions that are never validated (no ground truth comes back), the model cannot improve and you cannot measure whether it is working.

The problem is explainability-critical with no tolerance for errors In high-stakes regulated decisions (loan denial, medical diagnosis in some jurisdictions), a black-box model that cannot explain its reasoning may be legally unacceptable. A well-designed rule tree may be the right answer.

The decision boundary is simple and you have enough domain knowledge to encode it If fraud detection can be captured as "flag transactions > $5000 from a new country with a card added in the last 24 hours," that rule may outperform an ML model on a small dataset and is far more debuggable.

The payoff does not justify the cost An ML model that improves conversion by 0.1% may not be worth the engineering cost of building and maintaining a training pipeline, feature store, model serving infrastructure, and monitoring system.

Q9: Build vs buy vs use a pre-trained model — how do you make this decision?¶

Show answer

This is a practical engineering and economics question, not a pure ML question. The right answer depends on your timeline, team size, data ownership, and the specificity of your problem.

Use a pre-trained model (off-the-shelf) - The task is general enough that a pre-trained model addresses it well: language understanding (BERT, GPT), image classification (ResNet, EfficientNet), speech recognition (Whisper) - You have limited training data - Speed-to-production is the priority - Cost: inference pricing, no customisation, potential data privacy issues (sending data to a third-party API) - When this works: sentiment analysis, named entity recognition, image tagging, translation

Fine-tune a pre-trained model - The task is domain-specific enough that a general model underperforms but you have some labelled data - You want the benefit of pre-training (convergence, generalisation) with domain adaptation - Cost: requires a training infrastructure, but far less data and compute than training from scratch - When this works: medical text understanding, legal document classification, code completion for a specific language or framework

Build from scratch - Your domain is so specific that pre-trained models are useless or actively misleading (unusual feature spaces, proprietary data distributions) - You need full control over the architecture, training data, and model behaviour - You have sufficient labelled data and compute - Cost: highest investment, highest ceiling - When this is necessary: highly specialised recommender systems, proprietary signal processing, tasks with regulatory constraints on external data

Buy (third-party ML platform) - You need a solution quickly and your team lacks ML expertise - The task is commodity enough that a SaaS solution exists (fraud detection APIs, churn prediction platforms) - Cost: vendor lock-in, limited customisation, ongoing subscription cost

The decision framework: Start at the top of this list. Only move to a more expensive, complex option when the simpler option demonstrably fails to meet the success criteria.

Q10: How do you handle the cold start problem in a recommendation system?¶

Show answer

The cold start problem occurs when a recommendation system has insufficient data about a new user, new item, or new context to make relevant recommendations. It is one of the hardest operational problems in production recommendation systems.

Three types of cold start:

New user cold start The system knows nothing about the user's preferences.

Strategies: - Onboarding flow: explicitly ask users for preferences (genres, brands, topics) before their first session. Spotify's onboarding asks for favourite artists. This primes the recommendation system. - Popularity-based recommendations: start with the most popular or trending items — likely to be broadly appealing until personal signals accumulate. - Demographic/contextual inference: use what you can infer without history — location, time of day, device type, referral source — as initial signals. - Transfer from similar users: find users with similar onboarding responses and use their early behaviour as a proxy.

New item cold start A new item has no engagement data, so collaborative filtering cannot rank it.

Strategies: - Content-based bootstrapping: use item features (description, category, tags, metadata) to find similar items with known engagement profiles. Recommend the new item to users who liked the similar items. - Forced exploration: randomly inject new items into recommendations for a fraction of users and observe engagement. This is A/B testing at the item level. - Publisher/creator signals: if the item comes from a creator with known audience demographics, use the creator's historical audience as a prior.

New context cold start A new use case or product surface where no historical data exists.

Strategies: - Cross-domain transfer: if you have a music recommendation model, some of its learnings about user preferences transfer to podcast recommendations. Build on existing user representations. - Rule-based baseline: deploy a curated editorial selection while the system accumulates data. Replace gradually as engagement data grows.

The key principle: cold start is a data accumulation problem. Every strategy buys time by using less data-hungry signals until the system can collect the interactions needed for collaborative filtering. The goal is to provide acceptable recommendations immediately and improve automatically as data accumulates.

Q11: A classification model achieves 98% accuracy on the test set. Should you deploy it?¶

Show answer

Not necessarily — and this question is specifically designed to catch candidates who conflate high accuracy with good performance.

Step 1: Check the class distribution - If 98% of your test data belongs to the negative class, a model that predicts "negative" for every input achieves 98% accuracy and is completely useless. - Always check the baseline accuracy: what does the majority-class classifier achieve? If the baseline is 97%, a 98% model is marginally better than doing nothing.

Step 2: Look at the confusion matrix - What is the precision and recall on the minority (positive) class? - A model with 98% accuracy can have 0% recall on the class that matters most.

Step 3: Check the calibration - Does the model's predicted probability match the actual frequency of positive cases? - A model that outputs 0.9 probability for positives should be right about 90% of the time for those cases. - Calibration matters for any downstream system that uses probabilities for decision-making (threshold setting, risk scoring).

Step 4: Check the training/test split integrity - Is the 98% from data leakage? If any test examples influenced training (through feature engineering, scaling, or overlapping time windows), the number is inflated.

Step 5: Check the operational context - Is 98% accuracy good enough for the use case? In medical diagnosis, 98% may mean millions of incorrect diagnoses at scale. In spam filtering, it may be acceptable.

The correct answer in an interview: never evaluate a model by a single number. The first question is always "compared to what baseline, and on which metrics that matter for the business?"

Q12: How do you decide which features to include in an ML model?¶

Show answer

Feature selection is a process of maximising predictive signal while minimising noise, complexity, and the risk of leakage or unfairness.

Start with domain knowledge Before touching the data, list the features that should matter based on your understanding of the problem. In churn prediction: recency of last session, frequency of usage, depth of feature adoption, support ticket history, billing events. Domain knowledge is faster and more reliable than algorithmic selection for an initial feature set.

Evaluate features empirically on held-out data

Univariate analysis: what is the correlation (or mutual information for non-linear relationships) between each feature and the target? Features below a threshold can be dropped without loss.
Feature importance from a trained model: tree-based models provide impurity-based importance; use permutation importance on a validation set for a more reliable estimate.
SHAP values: model-agnostic feature importance that shows the contribution of each feature to each prediction, not just an aggregate.

Filter out problematic features - High cardinality categoricals with no meaningful encoding: raw user IDs, URLs, raw text without processing - Near-constant features: features that take one value for 99%+ of samples provide no signal - Features highly correlated with other features: in linear models, multicollinearity inflates variance; in tree models, correlated features split importance without improving performance - Features with high missingness: if >40% of values are missing and the missingness pattern is informative rather than random, the feature should be handled carefully or dropped

Final check: inference time availability Review every feature in your candidate set and confirm it will be available at prediction time in production. Drop any that require data that won't exist when the model needs to run.

Q13: What is the difference between optimising an offline ML metric and achieving a business outcome?¶

Show answer

This is one of the most important distinctions in applied ML, and failing to articulate it clearly is a common signal of inexperience.

Offline metrics are computed on historical data during model development: AUC, RMSE, precision@K, NDCG. They measure how well the model fits the historical data.

Business outcomes are changes in real user behaviour that create value: conversion, retention, revenue, satisfaction.

Why they diverge:

Distribution shift: offline data reflects the past. The model will be deployed into a future where user behaviour, competition, and product context may differ.
Feedback loops: deploying a model changes the data distribution it encounters. A recommender model changes which items users see, which changes which items get engagement data, which changes future training labels. Offline metrics ignore this dynamic.
Counterfactual bias in training data: offline training data was generated by some other system (the previous model or manual rules). Training a new model on this data implicitly learns the old system's biases. Offline metrics don't penalise this.
The metric-outcome gap: improving NDCG by 2% does not guarantee improving streaming starts. The features being reordered may not be the bottleneck to conversion.

The practical implication: - Offline metrics are a necessary but insufficient condition for deployment - Always follow offline evaluation with an online A/B test that measures the business outcome - When offline and online metrics disagree, trust the online A/B test — it is the ground truth - Large offline improvements that show no online lift are a signal that the offline metric is mis-specified

Q14: How do you think about fairness when framing an ML problem?¶

Show answer

Fairness is not an afterthought — it is a framing-level consideration that shapes what you build, how you evaluate it, and whether you deploy it.

Step 1: Identify the affected groups Who does this model make decisions about? Are there demographic groups (age, gender, race, income, geography) for whom model errors might have systematically different consequences?

Step 2: Identify the type of fairness concern - Disparate impact: the model produces different outcomes for different groups, even if group membership is not a feature. Example: a credit scoring model trained on historical data may perpetuate historical discrimination even without using race as a feature. - Calibration across groups: if a risk model assigns 30% probability to group A, are 30% of group A members actually high-risk? Miscalibration affects one group more than another. - False positive / false negative parity: in a hiring screening model, false negatives (qualified candidates incorrectly rejected) may disproportionately affect under-represented groups.

Step 3: Measure fairness alongside performance metrics Standard evaluation metrics (AUC, F1) aggregate across all users — they can hide significant disparities between subgroups. Always disaggregate metrics by protected attributes before deployment.

# Disaggregate performance by group
for group in ['group_a', 'group_b']:
    mask = X_test['group'] == group
    group_auc = roc_auc_score(y_test[mask], y_pred[mask])
    group_fpr = ... # false positive rate for this group
    print(f"{group}: AUC={group_auc:.3f}, FPR={group_fpr:.3f}")

Step 4: Understand the tradeoff It is mathematically impossible to simultaneously satisfy all common fairness definitions (demographic parity, equalised odds, calibration) when base rates differ between groups. You must decide — with stakeholders and legal counsel where appropriate — which fairness criterion matters most for the specific use case.

The interview answer quality signal: naming specific fairness definitions (not just "we should be fair"), showing awareness of the tradeoffs between them, and knowing that fairness evaluation requires disaggregated metrics.

← Previous: Experiment Design | → Next: Business Case Walkthrough