Model Serving¶
Taking a trained model from a notebook to a production system that reliably handles millions of requests is a discipline of its own. Interviewers in ML engineering and senior data science roles use these questions to assess whether you understand the gap between experimentation and production.
Q1: What is the difference between a trained model and a deployed model?¶
Show answer
A trained model is an artefact — a set of weights, coefficients, or tree structures stored in memory or on disk. A deployed model is a system that accepts requests, applies those weights to produce predictions, and returns results within a latency and reliability contract.
The gap between the two involves:
- Serialisation: the model must be saved in a format that can be loaded in the serving environment (which may differ from the training environment in language, OS, library versions)
- Preprocessing: the serving system must apply the exact same transformations to inputs that were applied during training — no exceptions
- Interface: the model needs an API surface — REST endpoint, gRPC service, or a batch job — that callers can use
- Reliability: a serving system needs health checks, retries, timeouts, and graceful degradation when the model fails
- Observability: logs, metrics, and traces so you know what predictions were made, how fast, and whether errors occurred
- Versioning: multiple model versions need to coexist so you can test new versions without immediately replacing the production version
Candidates who only think about the model artefact are thinking like researchers. Candidates who think about all of the above are thinking like engineers.
Q2: When do you use REST vs gRPC for model serving?¶
Show answer
REST (HTTP/JSON): - Simple to implement, debug, and test — curl and Postman work out of the box - Universally supported — any language, any platform - JSON is human-readable but verbose; parsing is CPU-intensive at high throughput - Typical overhead: 1–5ms for serialisation/deserialisation at moderate payload sizes - Use when: the client is a web browser, a third-party system, or interoperability with diverse callers matters
gRPC (HTTP/2 + Protocol Buffers):
- Binary serialisation — significantly smaller payloads and faster parse times than JSON
- Strongly typed contracts via .proto files — schema mismatches fail at compile time, not at runtime
- Bidirectional streaming support — useful for streaming inference or batching
- Typically 5–10x lower serialisation overhead than REST/JSON at high throughput
- Use when: internal service-to-service communication, latency is critical, payload sizes are large (image tensors, embeddings), or you need streaming
Practical rule: - External-facing endpoints where clients are diverse → REST - Internal microservice calls where latency and throughput matter → gRPC - Many production ML systems use gRPC internally (TensorFlow Serving, Triton Inference Server both use gRPC natively) and expose a REST gateway for external callers
What interviewers want to hear: you understand the tradeoff is fundamentally about serialisation cost and client compatibility, not just "gRPC is faster."
Q3: What are the main model serialisation formats and when do you use each?¶
Show answer
pickle / joblib (Python-specific): - Standard Python serialisation — works for any Python object including sklearn pipelines - Not safe across Python versions or library versions; a model pickled with sklearn 1.0 may not load in sklearn 1.3 - Never unpickle untrusted data — it executes arbitrary code - Use for: internal use cases where the training and serving environment are identical and controlled
ONNX (Open Neural Network Exchange):
- Framework-agnostic intermediate format — export from PyTorch or TensorFlow, run with ONNX Runtime in any language
- ONNX Runtime often provides faster CPU inference than the original framework (better kernel optimisation)
- Supports sklearn models via sklearn-onnx
- Use for: cross-framework portability, when serving language differs from training language, or when you want CPU inference optimisation
TorchScript:
- PyTorch-native serialisation — compiles a model to a graph that can run without Python
- Required for deploying PyTorch models in C++ serving environments or mobile
- Two modes: torch.jit.trace (fast, works for static input shapes) and torch.jit.script (handles dynamic control flow)
TensorFlow SavedModel: - TensorFlow's native serialisation — includes the computation graph, weights, and serving signatures - Required for TensorFlow Serving (the official TF model server) - The preferred format for TF/Keras models
Practical rule: for sklearn/XGBoost on a Python serving stack, joblib is fine for internal use. For cross-language, cross-framework, or performance-sensitive serving, prefer ONNX.
Q4: What is the tradeoff between batch inference and online inference from a latency/throughput perspective?¶
Show answer
Online inference: - One request arrives → model runs immediately → prediction returned - Optimised for low latency: target < 100ms for most applications, < 30ms for high-frequency use cases - Low throughput per compute unit — the model runs once per request, often with a batch size of 1 - GPUs are underutilised at batch size 1 — most of the GPU sits idle between matrix multiply operations - Infrastructure: always-on serving instances, autoscaling on QPS
Batch inference: - Collect a large set of inputs → run the model on all of them in one job → store predictions - Optimised for throughput: process millions of records per hour - Higher GPU utilisation — large batch sizes keep the GPU busy - High latency to any individual prediction — but acceptable because predictions are pre-computed - Infrastructure: scheduled jobs (cron, Spark, Airflow), no always-on serving instances needed
The GPU batch size effect: On a GPU, running a batch of 64 inputs takes nearly the same time as running 1 input — the GPU parallelises across the batch. At batch size 1, you pay the full GPU overhead for one prediction. This is why online GPU serving uses async batching (collect multiple requests, batch them together, run one forward pass, return individual results).
Latency vs throughput at different batch sizes (approximate, transformer model):
| Batch size | Latency (ms) | Throughput (predictions/sec) |
|---|---|---|
| 1 | 20 | 50 |
| 16 | 35 | 450 |
| 64 | 80 | 800 |
| 256 | 250 | 1,000 |
Online serving targets small batch sizes for latency. Batch inference targets large batch sizes for throughput.
Q5: What is shadow mode deployment and when do you use it?¶
Show answer
Shadow mode (also called shadow deployment or dark launch) runs a new model in parallel with the production model, receiving the same inputs, but its predictions are not used to make decisions. Instead, predictions are logged for offline analysis.
Why it is valuable: - You can measure the new model's behaviour on real production traffic without any risk to users - Catches bugs in the serving path (serialisation issues, preprocessing mismatches, latency regressions) before they affect real decisions - Lets you compare the distribution of predictions between old and new model before committing to a change - Identifies cases where the models disagree — the most informative examples for understanding what has changed
When to use it: - When the model change is significant enough that you want offline validation before any live traffic exposure - When the business cost of a bad prediction is high (fraud, medical, financial) - When you want to measure latency of a new model under real load before routing traffic to it
What it does not tell you: - Shadow mode cannot measure the business impact of the new model's predictions — no user behaviour is driven by shadow predictions. For that, you need an A/B test.
Typical workflow: shadow mode → A/B test on a small traffic slice → canary rollout → full rollout.
Q6: How do you A/B test models in production?¶
Show answer
A/B testing a model means routing a fraction of live traffic to the new model (the challenger) while the remainder continues to use the current model (the champion), then measuring the difference in a business metric.
Setup:
- Split users (not requests) randomly into control and treatment groups — user-level assignment ensures a consistent experience
- Use a hash of the user ID to assign groups deterministically: group = hash(user_id + experiment_id) % 100 < treatment_pct
- Start with 5–10% treatment traffic, expand after confirming no regressions
Metric selection: - Primary metric: the business metric the model is designed to move (CTR, conversion rate, churn rate) - Guardrail metrics: metrics you must not harm even if the primary metric improves (latency p99, error rate, revenue per user) - Define all metrics and their significance thresholds before the experiment starts — not after
Statistical validity: - Run the experiment for a full business cycle (minimum 1 week, preferably 2) to avoid day-of-week effects - Use a two-sample t-test or Mann-Whitney U test depending on your metric distribution - Target 80% statistical power at a 95% confidence level — use a sample size calculator to determine how long to run - Don't peek at results and stop early when you see significance — this inflates false positive rates (use sequential testing methods if you need early stopping)
What separates a good answer: knowing that you need guardrail metrics, not just the primary metric, and knowing the danger of peeking at results before the experiment is complete.
Q7: What is a canary deployment and what triggers a rollback?¶
Show answer
A canary deployment gradually shifts production traffic from the old model version to the new one, in stages, while monitoring for regressions at each stage.
Typical rollout stages: - 1% → 5% → 25% → 50% → 100% - Each stage runs for a defined duration (e.g. 30 minutes to 24 hours depending on traffic volume) - Automatically advance to the next stage if metrics are healthy; roll back if they are not
Rollback triggers (automatic): - Error rate > X% (e.g. model server returning 5xx errors at > 0.5%) - Latency p99 exceeds the SLA threshold (e.g. > 200ms) - Prediction null/NaN rate > 0.1% (model producing invalid outputs) - Business metric guardrail violated: CTR drops > 2% in the canary group relative to baseline
Rollback triggers (manual): - On-call engineer sees anomalous patterns that don't yet trigger automatic thresholds - A bug is identified in the new model's serving code
Rollback mechanics: - Keep the previous model version deployed and idle, ready to receive traffic - A rollback is just a traffic percentage change: set new model to 0%, old model to 100% - This should take seconds, not minutes — automated traffic management (Kubernetes, Istio, AWS ALB weighted routing)
What makes this answer strong: specifying concrete rollback criteria with numbers, not just "monitor and roll back if things go wrong."
Q8: How do model caching and feature caching reduce serving latency?¶
Show answer
Caching is one of the highest-leverage latency optimisations in ML serving. Two distinct types:
Feature caching: - Pre-compute expensive features and store them in a fast key-value store (Redis, Memcached) - At serve time, look up pre-computed features by user or item ID instead of computing them on-the-fly - Reduces the "feature computation" portion of serving latency from tens of milliseconds to < 1ms - Works best for features that are stable over a time window (user's 7-day purchase count, item metadata) - TTL (time-to-live) controls staleness: a 1-hour TTL means features are at most 1 hour old
Prediction caching: - Cache the model's output (the prediction) for a given input, not just the features - On a cache hit: return the cached prediction without running the model at all - Effective when: many requests are for the same (user, context) pair within a short time window, or inputs are drawn from a small discrete set - Works well for: product page ranking (same product seen by many users), spell correction, entity classification on a fixed entity set - Cache invalidation is the hard part: when the model is updated, the cache must be flushed or keyed to the model version
import redis
import hashlib
import json
cache = redis.Redis(host="localhost", port=6379)
def get_prediction_with_cache(model, features: dict, ttl_seconds: int = 300):
cache_key = hashlib.md5(json.dumps(features, sort_keys=True).encode()).hexdigest()
cached = cache.get(cache_key)
if cached:
return float(cached)
prediction = model.predict([list(features.values())])[0]
cache.setex(cache_key, ttl_seconds, str(prediction))
return prediction
Tradeoff: caching introduces staleness. A user who just completed a purchase will still see recommendations based on their pre-purchase state until the cache expires. Choose TTL based on how quickly the underlying reality changes.
Q9: When do you scale horizontally vs vertically for inference?¶
Show answer
Vertical scaling (bigger machines): - Add more CPU, RAM, or GPU to a single instance - Simple to implement — no changes to architecture - Has a hard ceiling: the largest available instance type - Useful when the model is too large to split across instances, or when inter-instance communication overhead would dominate
Horizontal scaling (more machines): - Add more instances, each running the same model - Load balancer distributes requests across instances - Scales to arbitrary traffic — the standard approach for stateless serving - Requires the model to be stateless (most inference is stateless — same input always produces same output) - Works seamlessly with autoscaling: scale instances up when QPS rises, down when it falls
When to use each:
| Situation | Choice |
|---|---|
| Model fits in memory, QPS is the constraint | Horizontal |
| Model requires a GPU that must be fully utilised | Vertical first, then horizontal |
| Stateless inference, variable traffic | Horizontal + autoscaling |
| Model is too large for a single GPU (LLM sharding) | Vertical (multi-GPU) or tensor parallelism |
| Low QPS but high latency per request (large models) | Vertical |
In practice: most ML serving uses horizontal scaling of stateless serving instances behind a load balancer. Vertical scaling is used to choose the right instance type per model (how much GPU memory is needed), then horizontal scaling handles throughput.
Q10: How do you tune batch size for GPU inference?¶
Show answer
On a GPU, the optimal batch size is the largest batch that: - Fits in GPU memory (VRAM) - Meets the latency budget
Why batch size matters for GPUs: GPUs are massively parallel processors designed for matrix operations. A single inference request (batch size 1) uses a tiny fraction of the GPU's compute capacity. Grouping requests into a batch lets the GPU process them in a single forward pass, dramatically increasing throughput without proportionally increasing latency.
Async (dynamic) batching: Rather than requiring callers to batch their requests, the serving system collects individual requests arriving within a short time window (e.g. 5ms) and groups them into a batch automatically. This is what NVIDIA Triton Inference Server calls "dynamic batching."
Tuning process: 1. Profile the model: measure latency and throughput at batch sizes 1, 2, 4, 8, 16, 32, 64, 128 2. Plot the latency-vs-throughput curve 3. Find the batch size where throughput plateaus — beyond that, you're adding latency without gaining throughput 4. Confirm the chosen batch size fits within your latency SLA
Memory constraint:
For a model with 4GB weights on a 16GB GPU, you have ~12GB for activations. A transformer processing 512-token sequences may need 50MB per sample, giving a max batch size of ~240.What to watch: GPU memory fragmentation can make the effective max batch size lower than the theoretical maximum. Use profiling tools (NVIDIA Nsight, torch.cuda.memory_summary()) to measure actual utilisation.
Q11: Why does p99 latency matter more than average latency?¶
Show answer
Average latency tells you about the median experience. The 99th percentile (p99) tells you about the worst experience for 1 in 100 users — and that 1% is often the most important.
Why p99 is the right operational target: - A service with average latency of 50ms and p99 of 2000ms is broken for 1% of users. In a system handling 10,000 requests per second, that's 100 users per second experiencing a 2-second response. At that scale, this is a serious UX and business problem. - For systems where requests are chained (microservices that call other microservices), the p99 of the chain is much worse than the p99 of any individual service. If you have 5 services each with p99 = 100ms, a request touching all 5 has p99 ≈ 500ms in the worst case. - High p99 often indicates a systematic problem: garbage collection pauses, memory swaps, cold start on serverless, lock contention — problems that don't show up in averages.
What to monitor: - p50 (median): typical user experience - p95: the tail — 5% of users are slower than this - p99: the extreme tail — used for SLA targets - p99.9 (sometimes p999): for critical systems (payments, healthcare), the 1-in-1000 case matters
SLA example: "Our model serving SLA is p99 < 150ms with 99.9% availability." This is a concrete, measurable contract — not "fast and reliable."
In interviews: citing percentile latencies instead of just average latency is a strong signal of production experience.
Q12: What does a model versioning and rollback strategy look like in practice?¶
Show answer
Model versioning ensures you can always return to a previous state, audit what was deployed when, and safely test new versions without disrupting production.
Version naming: Use semantic versioning or timestamp-based versioning. A common pattern:
(model name + semantic version + date + short git commit hash)Storage: - Model artefacts stored in object storage (S3, GCS) with versioned paths - Model metadata (metrics, training data version, feature list, training date) stored in a model registry (MLflow, W&B Model Registry, SageMaker Model Registry) - Never overwrite a deployed model artefact — always write to a new path
Deployment state machine:
A model moves through these stages explicitly. Only one model is "Champion" at a time, but multiple versions coexist in the registry.Rollback procedure: 1. Identify the last known-good version in the model registry 2. Update the serving configuration to point to that version's artefact path 3. Restart or signal the serving instances to reload the model 4. Verify predictions return to baseline behaviour
This should take < 5 minutes. If it takes longer, your rollback procedure is too manual.
What a model registry stores per version: - Artefact location (S3 path) - Evaluation metrics (AUC, F1, MAE — on the evaluation dataset) - Training data version (DVC hash or Delta Lake snapshot timestamp) - Feature list and schema - Training code version (git commit) - Who deployed it and when
Without this metadata, "rollback to the previous model" is ambiguous — which previous model? Trained on what data? Evaluated how?