ML System Design¶

ML system design interviews assess whether you can translate a vague business problem into a complete, production-grade machine learning system. These questions reveal how you think about tradeoffs, not just whether you know the algorithms.

Q1: What does an interviewer expect you to cover in an ML system design interview?¶

Show answer

The interviewer is testing your ability to think end-to-end, not just model selection. A strong answer walks through all of these layers in order:

Problem framing — clarify the business goal, define the ML task, identify the target metric
Data — sources, volume, labelling, freshness requirements
Features — what signals matter, how to compute them, train/serve consistency
Model — choice, justification, tradeoffs vs alternatives
Serving — batch or online, latency budget, infrastructure
Evaluation — offline metrics, online metrics, A/B testing strategy
Monitoring — drift detection, retraining triggers, failure modes

What separates a good answer from a great one: great answers surface tradeoffs at each layer rather than just listing components. "We could use a neural network, but given our 50ms latency budget, a gradient boosted tree is more practical at this scale."

Q2: What clarifying questions should you ask before designing anything?¶

Show answer

Never start designing until you understand the constraints. The questions that matter most:

About the problem: - What is the business goal this system needs to move? (Revenue, engagement, churn, safety?) - What does success look like — and how is it measured today without ML? - Is this a new system or replacing an existing one?

About data: - How much labelled data exists? How is it labelled? - How fresh does the data need to be — real-time events or weekly snapshots? - Are there regulatory constraints on data use (GDPR, HIPAA)?

About constraints: - What is the latency budget? (10ms? 100ms? Batch overnight?) - What is the expected request volume? (100 QPS? 1M QPS?) - Does the model output need to be explainable to end users or regulators? - What is the engineering team's capacity — are we building from scratch or integrating with existing infra?

Asking these questions signals seniority. Jumping straight to "I'd use a transformer" signals inexperience.

Q3: What is the full ML pipeline, and what can go wrong at each stage?¶

Show answer

The pipeline from raw data to serving predictions:

Raw Data → Data Ingestion → Feature Engineering → Training → Evaluation → Serving → Monitoring

Stage-by-stage failure modes:

Data ingestion: schema changes break downstream, late-arriving events corrupt labels, upstream outages create gaps
Feature engineering: train/serve skew (features computed differently at training vs serving time), null rates spiking silently
Training: label leakage, data imbalance, wrong evaluation metric optimised
Evaluation: evaluating on non-representative data, optimising offline metrics that don't correlate with online metrics
Serving: model too slow for the latency budget, memory footprint too large, cold start on new users
Monitoring: model silently degrading because no alerts are configured, ground truth arriving too late to catch drift early

The most insidious failures happen at the boundaries between stages — especially between feature engineering and serving, which is the root cause of training-serving skew.

Q4: How do you choose the right model for a given constraint?¶

Show answer

Model selection is a constraint satisfaction problem. Ask these questions in order:

Latency first: - < 10ms: logistic regression, shallow trees, pre-computed scores - 10–100ms: gradient boosted trees (XGBoost, LightGBM), small neural networks - > 100ms: deep neural networks, ensemble stacks, transformer-based models

Data size second: - < 10k samples: regularised linear models, shallow trees — deep models will overfit - 10k–1M samples: gradient boosted trees are typically strongest - > 1M samples: deep learning starts to outperform, especially for unstructured data

Interpretability third: - Regulatory requirement (credit scoring, medical): logistic regression, decision tree, or post-hoc explanation (SHAP on tree models) - No requirement: use whatever performs best

Maintenance cost fourth: - A complex model that nobody on the team understands is a liability. Prefer the simplest model that meets your metric target.

Rule of thumb: gradient boosted trees win on tabular data most of the time. Start there, then justify complexity if you need more.

Q5: What is the difference between online inference and batch inference? When do you use each?¶

Show answer

Online (real-time) inference: - Model receives a request and returns a prediction within the latency budget (typically < 200ms) - Used when the prediction must reflect the most recent state of the world - Examples: fraud detection on a live transaction, personalised search ranking, ad click prediction - Requires low-latency feature retrieval — features must be pre-computed and stored in a fast key-value store (Redis, DynamoDB)

Batch inference: - Model runs on a large dataset on a schedule (hourly, daily, weekly) - Predictions are stored and looked up at serve time — no model runs during the request - Used when predictions don't need to be real-time and computing all predictions up-front is feasible - Examples: email campaign targeting (score all users at midnight), churn risk scores refreshed daily, product recommendations pre-computed per user

Tradeoffs:

Dimension	Online	Batch
Latency	Low (ms)	High (hours, acceptable)
Freshness	Up-to-the-second	Stale by hours or days
Infrastructure	Complex (serving infra)	Simpler (scheduled jobs)
Coverage	Only scores users who request	Can score all users up-front

Hybrid: many systems use batch for the heavy lifting (compute user embeddings nightly) and online for the final ranking step (re-rank candidates using real-time context).

Q6: How often should you retrain your model?¶

Show answer

Retraining frequency is determined by how fast the world changes relative to your model's tolerance for staleness.

Factors that push toward more frequent retraining: - High data velocity (news, social media, market prices — distributions shift in hours) - Feedback loops (model predictions affect user behaviour, which affects future labels) - Seasonal or cyclical patterns - Concept drift detected in monitoring

Factors that push toward less frequent retraining: - Stable underlying distribution (medical imaging, geological data) - High cost of retraining (expensive compute, long training jobs) - Regulatory requirement to audit each model version

Common retraining cadences in practice: - Real-time streaming update: contextual bandits, online learning (rare — complex to maintain) - Daily: fraud, ad ranking, news recommendation - Weekly: product recommendations, churn models - Monthly or on-trigger: risk scoring, NLP models where data is stable

The right answer is: monitor for drift, set a retraining trigger when performance drops below a threshold, and run scheduled retraining as a baseline safety net. Don't retrain more often than you need to — more retraining means more risk of introducing a regression.

Q7: What is a feature store and why does it exist?¶

Show answer

A feature store is a centralised system for storing, sharing, and serving ML features. It solves two problems that become painful at scale:

Problem 1: Train/serve skew When training, features are computed from historical data in a batch job. When serving, features must be computed in real time from live data. If the computation logic diverges — even slightly — model performance degrades silently. A feature store enforces a single definition of each feature used in both contexts.

Problem 2: Feature duplication across teams Without a feature store, Team A computes "user's 7-day purchase count" for their churn model. Team B computes the same thing for their recommendation model — different code, potentially different results, wasted compute. A feature store lets both teams share the same pre-computed feature.

What a feature store provides: - Offline store: historical feature values for training (typically a data warehouse or Parquet files) - Online store: low-latency feature retrieval for serving (typically Redis or DynamoDB, sub-10ms lookups) - Feature pipeline: the computation logic that keeps both stores in sync

Examples: Feast (open source), Tecton, Vertex AI Feature Store, AWS SageMaker Feature Store.

When you need one: when you have multiple teams building multiple models and feature reuse + consistency has become a visible problem. For a single team with one or two models, a feature store adds more overhead than value.

Q8: What is the cold start problem and how do you handle it in a recommendation system?¶

Show answer

The cold start problem occurs when the system has no historical data about a new user or a new item, making it impossible to personalise recommendations.

Three flavours: - New user cold start: no interaction history — can't compute user embeddings or collaborative filtering signals - New item cold start: item has no ratings, views, or purchases — won't appear in collaborative filtering results - New system cold start: the entire system is new, so there's no data for anyone

Handling new user cold start: - Onboarding survey: ask the user for explicit preferences (genre, interest category) - Use demographic or contextual signals available without history (location, device type, referral source) - Fall back to popularity-based recommendations — "trending this week" - Explore: treat new users as an exploration problem, show diverse items and learn from early clicks

Handling new item cold start: - Use content-based features: item metadata (category, description, author, price band) - Warm up with editorial or partner-curated placement - Bootstrap with similar items' embeddings using content similarity - Hybrid model: blend collaborative filtering (strong for established items) with content-based (strong for new items)

What interviewers want to hear: that you recognise cold start as an exploration/exploitation tradeoff, not a pure engineering problem, and that your solution degrades gracefully (falls back to something reasonable) rather than breaking.

Q9: What is the tradeoff between model complexity and serving cost?¶

Show answer

More complex models almost always cost more to serve. The dimensions of that cost:

Latency: a deep neural network with 100M parameters takes 50–200ms to run a forward pass on CPU. A logistic regression with 1000 features takes < 1ms. At 10,000 QPS, that difference determines your infrastructure bill.

Memory: large models (transformer-based recommenders, large embedding tables) may require GPUs or high-memory instances, which are 5–20x more expensive per instance than CPU instances.

Throughput: complex models process fewer requests per second per instance, so you need more instances to handle the same load.

Maintenance: complex models are harder to debug when something goes wrong. A team that doesn't understand the model's internals can't diagnose a sudden performance drop.

The practical framework: 1. Establish the minimum performance threshold your business requires 2. Find the simplest model that meets that threshold 3. Only add complexity if the performance gain justifies the serving cost increase

A gradient boosted tree that gets you 85% of the performance of a transformer at 1/50th the serving cost is the right choice for most production systems. Reserve expensive models for where they genuinely move the needle.

Q10: Walk me through designing a recommendation system end to end.¶

Show answer

Use the standard two-stage architecture:

Stage 1: Candidate Generation (Retrieval) - Goal: narrow millions of items down to ~100–1000 candidates fast (< 20ms) - Method: approximate nearest neighbour search over item embeddings (FAISS, ScaNN) - Embeddings trained via: matrix factorisation, two-tower neural network, or collaborative filtering - Cold start fallback: popularity-based retrieval for new users

Stage 2: Ranking - Goal: score the 100–1000 candidates and return the top K - Method: a richer model (gradient boosted tree or a shallow neural network) with many features - Features: user history, item metadata, context (time of day, device), interaction statistics - Training signal: clicks, purchases, watch-time — define positive/negative labels carefully

Feature store: - User features (embedding, 7-day activity) → pre-computed nightly, served from Redis - Item features (embedding, popularity stats) → updated hourly - Context features → computed at request time

Evaluation: - Offline: NDCG@K, Hit Rate@K, MRR - Online: CTR, conversion rate, session length, revenue per session - A/B test any major change before full rollout

Monitoring: - Track item coverage (are we over-recommending a small set of popular items?) - Track novelty and diversity (are recommendations becoming too narrow?) - Alert on sudden drops in CTR or recommendation volume per user

What separates a great answer: acknowledging that offline metrics (NDCG) don't always predict online metrics (CTR), and that you must run A/B tests. Also noting the feedback loop risk — if you only recommend what gets clicked, you create a filter bubble.

Q11: Walk me through designing a fraud detection system.¶

Show answer

Fraud detection is a real-time classification problem with severe class imbalance and an adversarial environment.

Problem constraints to clarify first: - What is the latency budget? (Card fraud must decide in < 300ms before the payment times out) - What is the cost of a false positive (blocking a legitimate transaction) vs false negative (approving fraud)? - Is there a human review queue for borderline cases?

Data: - Transaction features: amount, merchant category, location, device fingerprint, time of day - User history: average spend, usual merchants, velocity (transactions in last 1 hour, 24 hours, 7 days) - Network features: is the merchant flagged? Is the card recently issued? - Labels: chargebacks, fraud reports — note that ground truth arrives days after the transaction (label delay)

Model: - Gradient boosted trees (XGBoost/LightGBM) work well for tabular transaction data - Two-stage: a fast rule-based pre-filter (< 1ms) catches obvious fraud, then a model scores the rest - Treat as imbalanced classification: use precision-recall AUC, not accuracy

Serving: - Online inference — every transaction scored in real time - Velocity features computed from a streaming store (Redis with TTL-based counters) - Model served via low-latency REST or gRPC endpoint, target < 50ms

Feedback loop risk: - If the model blocks a transaction, that transaction is never completed — no chargeback data to learn from - This creates survivorship bias: the model never sees the fraud it successfully blocked - Mitigation: periodically allow a small fraction of flagged transactions through (with limits) to get ground truth; use counterfactual logging

Adversarial drift: - Fraudsters adapt to the model — patterns shift faster than in other domains - Retrain frequently (daily or triggered), monitor feature distributions, track emerging fraud patterns in the flagged-but-not-blocked data

Q12: Walk me through designing a search ranking system.¶

Show answer

Search ranking is a learning-to-rank problem: given a query and a set of candidate documents, order them by relevance.

Two-stage architecture (same pattern as recommendations):

Retrieval: - Inverted index (BM25 or TF-IDF) for keyword matching — fast, handles exact match - Dense retrieval: query embedding vs document embedding similarity (bi-encoder, e.g. a fine-tuned BERT) - Combine both: hybrid retrieval improves recall

Ranking: - Pointwise: score each document independently (regression/classification) - Pairwise: learn which of two documents should rank higher (RankNet, LambdaRank) - Listwise: optimise the entire ranking list (LambdaMART, which is what LightGBM's rank objective implements) - Features: BM25 score, document freshness, click-through rate on this query-document pair, user personalisation signals, document quality signals

Training data: - Click logs: implicit signal — clicks imply some relevance, but are noisy (position bias — top results get clicked more regardless of relevance) - Correct for position bias: use inverse propensity scoring or a dedicated unbiased learning-to-rank approach - Human relevance labels: expensive but high quality — use for evaluation at minimum

Evaluation: - Offline: NDCG@10, MRR, MAP - Online: Click-through rate, dwell time, zero-result rate, query reformulation rate (a user who reformulates probably didn't find what they wanted)

Personalisation layer: - After ranking, apply a light personalisation re-rank using user history - Separate model or a feature added to the ranker — depends on data sparsity per user

What makes this answer strong: knowing the difference between retrieval and ranking, knowing that click data has position bias and naming a way to correct for it, and naming both offline and online evaluation metrics.