Data Pipelines¶
Data pipelines are the unglamorous backbone of every ML system. Interviewers ask about them because most ML failures in production are data failures, not model failures — and senior candidates know this.
Q1: What is a data pipeline and why do they fail?¶
Show answer
A data pipeline is a sequence of steps that moves data from a source to a destination, applying transformations along the way. In ML contexts, this typically means: raw events → cleaned, validated, feature-engineered data → model training or serving.
Why they fail — the most common causes:
- Schema changes: an upstream team adds, removes, or renames a column. If your pipeline has no schema validation, it either crashes or silently produces wrong output.
- Volume spikes: a marketing campaign sends 10x normal traffic. Pipelines sized for average load fail under peak load.
- Late-arriving data: events that belong to yesterday arrive today (mobile apps with poor connectivity, distributed systems with clock skew). Windowed aggregations computed too early will be wrong.
- Upstream outages: a source system goes down for 2 hours. The pipeline must handle gaps without corrupting downstream state.
- Silent data quality degradation: the pipeline runs successfully but null rates double, or a distribution drifts. No alert fires because there's no error — just wrong data.
The key insight: most pipeline failures are silent. The pipeline reports success while producing output that will degrade model performance days later. That's why data quality checks — not just pipeline health checks — are mandatory.
Q2: What is the difference between ETL and ELT? Which is more common in modern data stacks?¶
Show answer
ETL (Extract, Transform, Load): - Transform data before loading it into the destination - Historically used with data warehouses that had limited compute (transformations ran on ETL servers) - Examples: Informatica, Talend, traditional SSIS pipelines
ELT (Extract, Load, Transform): - Load raw data into the destination first, then transform it in place - Modern cloud data warehouses (BigQuery, Snowflake, Redshift) have cheap, scalable compute — transformations run inside the warehouse using SQL - Tools: dbt (data build tool) is the canonical ELT transformation layer
Why ELT is now dominant: - Raw data is preserved — you can reprocess it when requirements change, without re-extracting from source - SQL transformations in the warehouse are version-controlled (dbt), testable, and lineage-tracked - No ETL server to maintain — warehouse scales automatically - Faster time to value: data analysts can write transformations in SQL rather than waiting for engineering
When ETL still makes sense: - Sensitive data that must be masked before it enters any shared system (PII, PHI) — transform before loading to ensure compliance - Transformations that are too complex or expensive to run in the warehouse (heavy NLP processing, model inference)
Q3: When do you use batch processing vs stream processing?¶
Show answer
Batch processing: - Process data in large chunks on a schedule (hourly, daily) - High throughput, high latency (results available after the batch job finishes) - Tools: Apache Spark, dbt, SQL on BigQuery/Snowflake - Use when: the downstream consumer can tolerate stale data and throughput matters more than freshness - Examples: nightly feature computation for a churn model, weekly reporting aggregations
Stream processing: - Process events as they arrive, continuously - Low latency (results available in seconds to milliseconds), lower throughput per unit compute - Tools: Apache Kafka (message queue), Apache Flink (stateful stream processing), Spark Structured Streaming - Use when: decisions must be made on current data — fraud scoring, live dashboards, real-time personalisation - Examples: computing a user's transaction velocity in the last 60 minutes for fraud detection
The practical decision framework:
| Question | Answer → Use |
|---|---|
| Can the prediction be hours old? | Batch |
| Does the feature need data from the last minute? | Streaming |
| Is the data volume > billions of rows/day? | Batch (Spark) |
| Do you need sub-second feature freshness? | Streaming (Flink) |
Hybrid (Lambda architecture): run a batch layer for historical accuracy and a streaming layer for real-time approximations. Merge results at query time. Operationally complex — prefer pure streaming (Kappa architecture) if your stream processor can handle the historical backfill.
Q4: What data quality checks should every ML pipeline include?¶
Show answer
Data quality failures are silent killers. These checks should run automatically after every pipeline execution:
Schema validation: - Column names and data types match the expected schema - No unexpected new columns (which might indicate upstream changes) - Required columns are present
Null rate checks: - Null rate per column should be within a historical baseline ± some threshold - A feature that is usually 2% null becoming 40% null is a serious signal - Alert at > 2x the historical null rate
Distribution checks: - For numerical features: mean, standard deviation, min, max compared to historical range - For categorical features: value counts — unexpected new categories or disappearing expected ones - Use Population Stability Index (PSI) for a formal distribution shift score
Volume checks: - Row count should be within expected range (± 20% of the rolling average is a common threshold) - A sudden 90% drop in rows usually means an upstream outage or join condition changed
Referential integrity: - Foreign keys resolve — a user_id in the events table should exist in the users table - Timestamp ordering — event timestamps shouldn't be in the future
Tools: Great Expectations (open source), dbt tests, Deequ (Spark-based, AWS), Monte Carlo, Bigeye.
# Great Expectations example
import great_expectations as ge
df = ge.read_csv("features.csv")
df.expect_column_values_to_not_be_null("user_id")
df.expect_column_mean_to_be_between("purchase_amount", min_value=10, max_value=500)
df.expect_column_proportion_of_unique_values_to_be_between("category", min_value=0.001, max_value=0.5)
Q5: How do you ensure consistency between feature engineering at training time and serving time?¶
Show answer
Training-serving skew is one of the most common causes of model degradation in production. It happens when the code that computes features during training differs from the code that computes features during serving.
Common causes: - Two separate codebases: a Python batch job for training, a Java service for serving - Different business logic applied at each stage - Different data sources — training uses warehouse data, serving uses an API with slightly different semantics - Bugs in one codebase that don't exist in the other
Prevention strategies:
1. Single source of truth for feature logic: Store feature definitions in a feature store (Feast, Tecton). Both the training pipeline and the serving layer read from the same definitions. The feature store handles computing features from both historical data and live data using the same code path.
2. Shared library: If a feature store is overkill, extract all feature transformation logic into a shared Python library. Both the training pipeline and the serving service import and use the same functions.
3. Log-and-replay: Log the exact feature vector sent to the model at serving time. For retraining, use these logged features rather than recomputing from raw data. Eliminates skew entirely — training uses the same features the model saw in production.
4. End-to-end tests: Write tests that feed identical raw inputs through both the training pipeline and the serving path, and assert that the output features are identical.
Q6: What is data versioning and why does it matter for ML?¶
Show answer
Data versioning means treating datasets the same way software engineers treat code: each version is immutable, identified by a hash or tag, and you can reproduce any past experiment by checking out the data version used at that time.
Why it matters for ML specifically:
- Reproducibility: to reproduce a model's results, you need the exact training data used, not "approximately the same data." Datasets change over time — rows get deleted (GDPR), labels get corrected, features get re-engineered.
- Debugging regressions: if a new model performs worse, was it the model change or the data change? Without data versioning, you cannot isolate the cause.
- Rollback: if a data pipeline bug corrupts a feature, you need to retrain from a known-good data snapshot.
- Audit: regulators may require you to show exactly what data was used to train a model that made a consequential decision.
Tools:
- DVC (Data Version Control): Git-like CLI for large files and datasets. Stores data in S3/GCS, tracks versions in Git metadata. dvc push, dvc pull, dvc repro are the core commands.
- Delta Lake / Apache Iceberg: table formats for data lakes that provide ACID transactions, time travel (query data as it looked at any past timestamp), and schema evolution.
- MLflow Artifacts: lighter-weight — logs datasets as artifacts alongside runs, so each experiment is linked to the data it used.
Q7: What is backfilling and why is it hard?¶
Show answer
Backfilling is the process of computing feature values or running a pipeline for historical time periods — typically because the pipeline is new, was fixed after a bug, or a new feature needs historical values for model training.
Why it sounds easy but isn't:
Point-in-time correctness: if you want to compute "user's purchase count in the last 30 days as of 2024-01-15", you need to know what data existed on that date — not what you know today. Users who were deleted, transactions that were refunded, records that were corrected — all of these change the ground truth retrospectively. A naive backfill using today's full dataset produces "future-aware" features that didn't actually exist at that historical point. This is a form of data leakage.
Volume: backfilling a year of daily features for 10 million users is a large compute job. It may saturate your data warehouse or source system.
Idempotency: the backfill job must be safe to run multiple times. If a partial run fails halfway through, re-running it from the start must produce the same correct output without duplicating data.
Dependencies: feature A depends on feature B depends on raw table C. All three must be backfilled in the correct order.
Best practices: - Store raw events with immutable timestamps — never overwrite historical records - Use a time-travel-capable table format (Delta Lake, Iceberg) so you can query the state of data at any historical point - Design pipeline logic to be idempotent: delete-then-insert (or use MERGE/UPSERT) for the target partition before writing - Backfill in small time chunks with checkpointing so failures are recoverable
Q8: How do you handle late-arriving data in a streaming pipeline?¶
Show answer
In distributed systems, events often arrive after their nominal event time. A mobile app records a click at 14:00:00 but the event doesn't reach the pipeline until 14:07:30 due to intermittent connectivity. A windowed aggregation that closed the 14:00–14:05 window at 14:05 missed that event.
Watermarks: Stream processors like Flink and Spark Structured Streaming use watermarks to define how long to wait for late events before closing a window. A watermark of 5 minutes means: "close the 14:00–14:05 window at 14:10, giving late events up to 5 minutes to arrive."
# Spark Structured Streaming: watermark + window
from pyspark.sql import functions as F
df.withWatermark("event_timestamp", "5 minutes") \
.groupBy(F.window("event_timestamp", "5 minutes"), "user_id") \
.count()
Tradeoff: a longer watermark means more completeness (fewer missed events) but higher latency (results available later). Choose based on how late events actually arrive in your system — measure the 99th percentile event delay in production.
Handling events that arrive after the watermark: - Drop: simplest — ignore events that arrived too late. Acceptable if late events are rare (< 0.1%) and the business impact is low. - Side output / dead-letter queue: route late events to a separate stream for manual review or later correction. - Recompute: trigger a batch recomputation of affected windows after a delay. Accurate but adds operational complexity.
The key insight interviewers want: you understand that event time ≠ processing time, and that any windowed aggregation needs an explicit policy for late data — not just an assumption that data arrives in order.
Q9: How do you handle schema evolution without breaking downstream systems?¶
Show answer
Schema evolution is inevitable: features get added, columns get renamed, data types get changed. Without a strategy, every change breaks consumers.
Safe changes (backward compatible): - Adding a new nullable column with a default value - Adding a new optional field in a schema registry
Unsafe changes (breaking): - Removing a column a downstream model depends on - Changing a column's data type (e.g. integer to string) - Renaming a column without an alias
Strategies:
Schema registry (Confluent Schema Registry, AWS Glue Schema Registry): Producers register schemas; consumers validate incoming data against the registry. Enforces compatibility rules — you can configure it to reject breaking changes before they reach consumers.
Versioned schemas: Maintain schema versions explicitly. When a breaking change is needed, create a new schema version and run both versions in parallel during a migration window. Consumers opt into the new version on their own schedule.
Additive-only discipline: Enforce a policy: never delete or rename columns. Instead, deprecate old columns (mark them in documentation) and add new ones. After all consumers have migrated, columns can be removed in a coordinated, announced deprecation cycle.
Contract testing: Consumers publish what fields they depend on (a "consumer contract"). Any schema change that violates a contract fails CI automatically — before the change reaches production.
Q10: What causes training-serving skew and how do you prevent it?¶
Show answer
Training-serving skew is the discrepancy between the features a model was trained on and the features it receives at serving time. It is one of the most common causes of silent production degradation.
Root causes:
- Different code paths: training uses a Spark batch job; serving uses a Java microservice. Same logic, different implementation — bugs accumulate.
- Different data sources: training reads from a data warehouse; serving reads from an operational database that hasn't been fully synced.
- Temporal leakage in training: training features are computed with data that wasn't available at the time the label was generated (future data leaking into training). The model looks great offline but fails online.
- Preprocessing mismatch: a scaler fit on training data isn't saved and applied at serving. The serving path applies a different scaler — or none at all.
Prevention: - Use a feature store with a single code path for both training and serving - Save all preprocessing transformers (scalers, encoders) alongside the model as part of a pipeline object. Never refit them at serving time. - Log features at serving time and periodically compare the distribution to training features - Write integration tests that run the same input through both the training and serving code paths and assert equality
# Save the full pipeline including preprocessing
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
import joblib
pipeline = Pipeline([
("scaler", StandardScaler()),
("model", GradientBoostingClassifier())
])
pipeline.fit(X_train, y_train)
joblib.dump(pipeline, "model_pipeline.pkl") # scaler and model saved together
# At serving time
pipeline = joblib.load("model_pipeline.pkl")
prediction = pipeline.predict(raw_features) # same scaler applied automatically
Q11: What is idempotency in pipelines and why does it matter?¶
Show answer
An idempotent pipeline produces the same output no matter how many times it runs with the same input. Running the job once or ten times leaves the system in the same state.
Why it matters: - Pipeline failures are normal. Retries are the standard recovery mechanism. If a retry produces duplicate rows or corrupts state, you have a much worse problem than the original failure. - Backfills need to be re-runnable safely. If you discover a bug and need to rerun a week of data, you want to overwrite — not append — the affected output. - Debugging is easier when reruns are deterministic.
How to achieve idempotency:
Delete-before-insert (partition overwrite): If your output is partitioned by date, delete the output partition before writing. If the job fails mid-write, the partition is missing (detectable) rather than partially correct (invisible corruption).
# PySpark: overwrite a specific partition
df.write \
.mode("overwrite") \
.partitionBy("date") \
.parquet("s3://my-bucket/features/")
UPSERT / MERGE: For databases, use an upsert (insert or update if key exists) rather than a plain insert. Reruns update existing rows rather than duplicating them.
Deduplication keys: If the downstream system is append-only (like a Kafka topic or a log), add a unique event ID. Consumers deduplicate on this ID.
What separates a good answer: explaining that idempotency is a design property, not something you achieve by accident — it requires explicit design of your write strategy.
Q12: What is data lineage and why is it operationally important?¶
Show answer
Data lineage tracks the origin and transformation history of every piece of data — where it came from, what transformations it went through, and what downstream systems depend on it.
Operational importance:
Debugging: a model's prediction is unexpectedly wrong. Lineage tells you which features were used, which pipeline computed them, and which raw tables those features came from. Without lineage, debugging requires manually tracing code across multiple systems.
Impact analysis: an upstream team wants to change the schema of a raw events table. Lineage shows you every downstream dataset, feature, and model that would be affected — before the change is made.
Compliance and audit: GDPR requires you to know where every piece of personal data lives. Lineage makes it possible to answer "show me every system that processed user ID X's data."
Incident response: a data quality issue is detected in production. Lineage lets you identify all models and dashboards consuming the corrupted data, so you can quarantine affected outputs.
Tools: - Apache Atlas: open source, integrates with Hadoop ecosystem - OpenLineage: open standard for lineage events, supported by Airflow, dbt, Spark - dbt: automatically generates lineage for all SQL transformations as a DAG - DataHub, Amundsen: data catalogues with lineage built in
What interviewers want to hear: that you treat lineage as a first-class operational requirement, not an afterthought — and that you know concrete tools rather than speaking only in abstractions.