Experiment Design¶
Experiment design questions test whether you can set up a valid causal test — not just run a statistical procedure. Interviewers want to know that you understand what can go wrong and how to prevent it, not just how to compute a p-value.
Q1: How do you design an A/B test from scratch? Walk through every decision you need to make.¶
Show answer
A complete A/B test design has eight decisions. Missing any of them is a risk to the validity of your results.
Decision 1: Define the hypothesis - What is the change you are testing? - What is the mechanism by which it should produce an effect? - What is the null hypothesis? (default: the change has no effect on the primary metric) - One-sided or two-sided test? Two-sided is the safe default unless you pre-commit to a direction.
Decision 2: Choose the primary metric - One metric that directly captures the value the change is meant to deliver - Must be measurable within the experiment window - Must be sensitive enough to detect the expected effect size (low enough variance)
Decision 3: Define guardrail metrics - What must not get worse when the primary metric improves? - Decide the acceptable bounds before seeing results
Decision 4: Choose the randomisation unit - User, session, device, request, or something else? - Rule: the randomisation unit must be at least as granular as the analysis unit - If the treatment affects how users interact with each other, randomising by user is insufficient — randomise by cluster
Decision 5: Calculate required sample size - Inputs: significance level (α), power (1−β), baseline metric value, minimum detectable effect (MDE), metric variance - Common defaults: α = 0.05, power = 0.80 - The MDE is a business decision, not a statistical one: what is the smallest improvement worth acting on?
Decision 6: Determine experiment duration - At minimum: time to reach required sample size - In practice: always run for at least one full week (7 days) to capture day-of-week variation - Consider novelty effect: if users might be unusually curious about the change initially, run longer (2–4 weeks) to reach steady state
Decision 7: Implement and validate before launch - Run an A/A test first: confirm that your experiment infrastructure assigns users correctly and that the control and treatment groups have no pre-existing metric differences - Verify that logging is working for all relevant events - Confirm that the treatment is actually being applied to the treatment group (no leakage)
Decision 8: Define the analysis plan before peeking - What test statistic will you use? - Will you apply any variance reduction (CUPED)? - How will you handle multiple metrics (multiple testing correction)? - What constitutes a "ship" decision vs "iterate" vs "kill"?
Deciding the analysis plan in advance prevents p-hacking — the temptation to try different analysis approaches until you find significance.
Q2: How do you calculate the sample size for an A/B test? What inputs do you need?¶
Show answer
Sample size calculation requires four inputs. Missing any one of them means your calculation is either wrong or based on an arbitrary assumption.
The four inputs:
1. Significance level (α) The probability of a false positive — declaring a winner when there is no real effect. Standard: α = 0.05.
2. Power (1 − β) The probability of correctly detecting a real effect when it exists. Standard: 0.80 (accepting 20% false negative rate). Higher stakes tests should use 0.90.
3. Baseline metric value The current value of your primary metric in production. For a proportion metric (conversion rate): the current conversion rate. For a continuous metric (revenue per user): the current mean and standard deviation.
4. Minimum detectable effect (MDE) The smallest improvement that would be meaningful enough to act on. This is a business decision, not a statistical one. The smaller the MDE, the larger the sample needed.
For a two-proportion test (e.g., conversion rate):
from statsmodels.stats.proportion import proportion_effectsize
from statsmodels.stats.power import NormalIndPower
baseline_rate = 0.08 # 8% current conversion rate
target_rate = 0.084 # 5% relative lift = 4pp absolute * 0.05 = 0.4pp
target_rate = 0.084 # 8.4% target (0.4pp absolute lift)
effect_size = proportion_effectsize(baseline_rate, target_rate)
analysis = NormalIndPower()
n_per_group = analysis.solve_power(
effect_size=effect_size,
alpha=0.05,
power=0.80,
alternative='two-sided'
)
print(f"Required per group: {n_per_group:,.0f}")
print(f"Total required: {2 * n_per_group:,.0f}")
What moves sample size and in which direction: - Larger MDE → smaller sample (easier to detect) - Higher variance in the metric → larger sample - Stricter α (e.g., 0.01) → larger sample - Higher power (e.g., 0.90) → larger sample - More variants beyond A/B → larger total sample (multiple testing)
Common mistake: calculating sample size after you see results to "check if the test was adequately powered." This is circular reasoning. Power analysis is a pre-experiment design tool.
Q3: What is the novelty effect, and how does it bias your experiment results?¶
Show answer
The novelty effect is the tendency for users to engage more with a new experience simply because it is new — not because it is better. This temporarily inflates the apparent effect size during the experiment window.
The reverse — change aversion — also exists: some users engage less with a new design because it disrupts familiar patterns, even if the new design is objectively superior. Both effects decay as users habituate.
Why it matters: if your experiment window captures only the novelty period, you will either overestimate or underestimate the feature's true sustained effect.
Signals that novelty effect is distorting your results: - The treatment metric shows a spike in the first 1–3 days, then gradual regression toward the control level - The lift is concentrated in infrequent users (curious, occasional explorers) rather than core users - When you segment by user tenure, new users show the lift but long-tenured users don't — or vice versa
How to handle novelty effect: - Run the experiment longer: 2–4 weeks instead of 1. The novelty decays; what remains is the genuine effect. - Analyse by day: plot the daily metric difference between treatment and control. If the gap is closing over time, novelty is present. - Segment by user tenure and frequency: restrict primary analysis to users who have used the feature area before (not first-time visitors). These users are less subject to novelty. - Long-term holdouts: after launch, maintain a holdout group on the old experience for 30–90 days to measure the sustained effect.
Interview answer quality signal: mentioning both novelty effect and change aversion, and knowing that the solution is a longer experiment window and longitudinal analysis — not more statistical power.
Q4: What is a network effect in the context of A/B testing, and when does it invalidate standard experiments?¶
Show answer
A network effect (or interference between units) in A/B testing occurs when one user's treatment assignment affects another user's outcome. This violates the Stable Unit Treatment Value Assumption (SUTVA) — the assumption that the potential outcome for any unit depends only on that unit's own assignment, not on anyone else's.
When SUTVA is violated, your experiment's estimated treatment effect is biased.
Where interference is common:
- Social networks: if user A gets a new feed algorithm and posts more content, user B (in control) sees more of A's content — the control group is contaminated
- Two-sided marketplaces (Uber, Airbnb, DoorDash): giving a discount to some riders (treatment) reduces available drivers for all riders, including those in control. The control group's experience is degraded, making the treatment look better than it is
- Ride-sharing dispatch: a new routing algorithm for treatment drivers changes wait times for all riders
- Multiplayer games: matchmaking changes affect every player in a match, regardless of their own assignment
The consequence: standard A/B tests produce a biased estimate of the global average treatment effect (GATE). In marketplaces, the bias is almost always in the direction of overestimating the treatment's benefit (control is artificially degraded).
Alternatives when interference exists: - Cluster-level randomisation: assign entire cities, markets, or social communities to treatment or control. Interference stays within clusters. Trade-off: fewer randomisation units → less statistical power. - Switchback experiments: alternate the system between treatment and control over time windows within the same market (see Q6) - Graph-cluster randomisation: for social networks, identify communities and randomise at the community level - Bipartite experiment designs: used by LinkedIn, DoorDash for two-sided marketplace experiments
Q5: What should you do when you cannot randomise by individual user?¶
Show answer
User-level randomisation is ideal but often impossible: in marketplaces, social networks, shared systems, or when the treatment is applied at a system level (a new dispatch algorithm affects everyone).
Alternative designs:
Geo experiments (market-level randomisation) - Assign entire geographic markets (cities, countries, DMA regions) to treatment or control - Analyse market-level outcomes and aggregate - Requires: matching markets on pre-experiment trends; enough markets for statistical power (typically 30+ is comfortable, fewer is hard) - Used by: Google, Facebook for ads experiments; Uber for pricing algorithm tests
Switchback experiments (time-based randomisation) - Alternate treatment and control over time windows within the same market (e.g., flip every 30 minutes) - Eliminates cross-unit interference because all users in the market are in the same state at the same time - Requires: short enough periods that carryover effects are minimal; washout periods between flips if carryover is a concern - Statistical analysis: regression with market and time fixed effects - Used by: Lyft, DoorDash, Instacart for dispatch and pricing experiments
Synthetic control - Construct a weighted combination of untreated markets that closely matches the treated market's pre-period trajectory - Use the synthetic control as the counterfactual post-treatment - Requires: multiple untreated units; long pre-period; stable pre-period trend
Difference-in-differences (DiD) - Compare the change in the treated group's metric to the change in an untreated group's metric over the same period - Assumes: parallel trends — treated and control would have moved the same way absent the treatment - Key diagnostic: check that treatment and control trends were parallel before the treatment
Practical note: when explaining a design to a non-technical stakeholder, focus on the intuition — "we are comparing markets that got the change to markets that didn't, during the same time period" — not the technical machinery.
Q6: What are holdout groups, and how are they different from control groups in an A/B test?¶
Show answer
Both are groups of users who receive the existing (old) experience rather than a new treatment. The difference is timing and purpose.
Control group in an A/B test: - Runs concurrently with the treatment during the experiment window (1–4 weeks) - Used to estimate the causal effect of the change in the short term - Dissolved when the experiment ends — all users eventually get the winning variant
Holdout group (holdback group): - A small fraction of users (typically 1–10%) kept on the old experience after the feature ships to the general population - Continues for weeks or months post-launch - Measures the long-term, sustained effect of the change — including effects that the A/B test window was too short to capture
What holdouts measure that A/B tests miss: - Novelty effect decay: does the lift persist once novelty wears off? - Network and ecosystem effects: as more users adopt the feature, does the product-level metric change in ways the unit-level A/B test couldn't show? - Habit formation: some product changes only generate value once users have changed their behaviour patterns — this takes weeks, not days
Costs of holdouts: - You are deliberately withholding a potentially better experience from a subset of users — ethical and business trade-off - Holdout users may notice inconsistencies (e.g., they see shared content from feature-enabled friends but cannot access the feature themselves) - Operationally complex at scale
When to use holdouts: for significant feature launches where the A/B test was short, where novelty effect is suspected, or where the business impact is large enough to justify the operational overhead.
Q7: Why are some metrics hard to move in a short experiment, and how do you handle this?¶
Show answer
Metric sensitivity is determined by signal-to-noise ratio. Metrics with high variance relative to the expected effect size require large sample sizes — and long experiments — to detect movements reliably.
Reasons a metric is hard to move in a short window:
High variance Revenue per user has high variance because some users spend a lot and most spend nothing. The variance in this metric is driven by the heavy tail of high spenders. A 2-week experiment may not accumulate enough data to reliably detect a 5% lift in ARPU.
Rare events Conversion events (purchases, sign-ups) that happen at rates below 1–2% require very large sample sizes to measure any change in rate.
Slow-responding metrics Retention (D30, D90) cannot be measured in a 2-week experiment — you need to wait 30 or 90 days to see whether users came back.
Approaches to handle low-sensitivity metrics:
1. Use a more sensitive leading indicator as the primary metric Instead of D30 retention (hard to move), use a leading indicator like "users who complete their second core action" (easier to measure, predicts D30).
2. Variance reduction (CUPED) Use pre-experiment data on the same users to control for individual-level variation. Can reduce metric variance by 20–50%, effectively increasing statistical power without adding users.
3. Longer experiment duration More data reduces the standard error of the mean. Doubling the experiment length reduces uncertainty by 1/√2 ≈ 30%.
4. Stratified sampling Ensure treatment and control have balanced compositions on high-variance segments (power users, high spenders). Reduces variance from compositional differences.
5. Accept a larger MDE If you cannot afford the sample size to detect a 1% lift, set the MDE to 5%. You will miss smaller effects, but you will confidently detect large ones.
Q8: You run an A/B test and get a "flat" result — p-value of 0.4. What does this actually mean?¶
Show answer
A flat result (p > 0.05) is not evidence that the change had no effect. This is the most important and most commonly misunderstood result in A/B testing.
What a p-value of 0.4 actually means: - If the null hypothesis is true (no real effect), observing a difference as large as you measured by chance has a 40% probability - This is consistent with: (a) no real effect, (b) a real effect that your test was underpowered to detect
The critical question: was your test adequately powered? - If you pre-calculated sample size for 80% power at your MDE, and ran until you reached that sample size, a flat result is meaningful — you can reasonably conclude the effect is smaller than your MDE (though not zero) - If you stopped early, or if you never calculated sample size, a flat result is nearly uninterpretable — you may have simply run out of time or data
How to communicate a flat result correctly: - "We did not detect a statistically significant effect. Given our sample size and the assumed MDE of X, we had 80% power to detect an effect of this size or larger. This result is most consistent with either no real effect or an effect smaller than X." - Not: "The change didn't work" — this implies you proved the null - Not: "We need more data" — if you reached your planned sample size, you have the data you planned for
When a flat result should lead to more experimentation: - If your MDE was larger than the minimum business-relevant effect - If the experiment had technical issues during the run - If the metric showed high variance that wasn't accounted for in planning
The professional answer: interpret a flat result in terms of the confidence interval, not just the p-value. "The 95% confidence interval for the lift is [−2%, +3%]. We can rule out effects larger than 3% in either direction."
Q9: When should you stop an experiment early, and what are the risks?¶
Show answer
There are exactly two valid reasons to stop an experiment before it reaches its planned sample size.
Valid reason 1: A guardrail metric shows clear, significant harm If the treatment is causing measurable damage to a critical metric (crash rate, churn, revenue decline), stopping is a business and ethical obligation. This is not about statistical significance — it is about preventing ongoing harm to users.
Valid reason 2: You are using a sequential testing framework that explicitly permits early stopping Methods like alpha spending functions (O'Brien-Fleming boundaries), always-valid p-values, or the mixture sequential probability ratio test (mSPRT) allow you to look at results multiple times and stop early with valid error guarantees. The catch: these methods require that you pre-commit to the sequential framework before the experiment starts — they are not a retroactive justification for peeking.
Why stopping early for a winning result is problematic: - Any experiment's p-value will dip below 0.05 eventually by chance, even under the null - If you stop when you see p < 0.05 without sequential testing controls, your actual Type I error rate is much higher than 5% - Simulations show that continuous monitoring and stopping at first significance can produce effective false positive rates of 25–40%
What to do instead when you are tempted to stop early: - Pre-commit to the duration and sample size - If results look good early, resist — the expected value of waiting is almost always positive - If you must look early (operational constraints), use an alpha spending function with a pre-specified interim look
The subtle trap: stopping early for a negative result is equally problematic. "It looks flat after one week, let's kill it" — but you planned for two weeks. You are now underpowered and calling a non-result when you may simply have insufficient data.
Q10: What is the peeking problem, and how do you avoid it?¶
Show answer
The peeking problem is the inflated Type I error rate that results from making stopping decisions based on interim looks at experiment results — even if you don't formally stop early.
The mechanics: Under the null hypothesis, the p-value is a uniform random variable that fluctuates over time as data accumulates. At any single pre-specified point in time, the probability that p < 0.05 is exactly 5%. But if you monitor the p-value continuously and are willing to act when it crosses 0.05, the probability that it crosses 0.05 at some point during the experiment is much higher — often 20–40% over a typical experiment window.
Why this happens: each additional look at the data is an additional chance to observe a spurious crossing of the threshold. The p-value is not independent across looks.
Concrete example: running a 4-week experiment and checking the dashboard every day, with a willingness to call the experiment if p < 0.05 at any check, can inflate the false positive rate from 5% to over 25%.
Solutions:
1. Fixed-horizon testing: look once, at the end Commit to a sample size and look at results exactly once. Simple and most commonly taught — but inflexible in practice.
2. Pre-specified interim analyses with alpha spending If you must look multiple times, decide in advance how many looks you will take and apply an alpha spending function (O'Brien-Fleming is conservative; Pocock is less so). Each interim look gets a fraction of your total α budget.
3. Sequential testing / always-valid inference Methods like mSPRT or confidence sequences allow continuous monitoring with statistical guarantees. The "p-value" (or e-value) remains valid at any stopping time. Used by Spotify, Netflix, and others at scale.
4. Bayesian A/B testing Posterior probabilities update correctly as data arrives and are interpretable at any point. Does not have a peeking problem in the frequentist sense — though Bayesian decisions based on the posterior can still be suboptimal if made too early.
Q11: What is stratified sampling in A/B tests, and why does it reduce variance?¶
Show answer
Stratified sampling (or stratified randomisation) divides the user population into homogeneous subgroups (strata) before randomisation, then assigns users to treatment and control within each stratum — ensuring that each stratum is equally represented in both groups.
Why it reduces variance: In a standard random assignment, the treatment and control groups may end up with slightly different compositions purely by chance — more power users in treatment, more new users in control, for example. This compositional imbalance adds noise to your metric estimate.
Stratified randomisation ensures that the treatment and control have identical proportions of each stratum. The metric estimate then has less variance because the compositional noise has been removed by design.
Common strata for stratified randomisation: - User tenure (new vs returning) - Platform (iOS, Android, Web) - Geography (country or region) - Activity level (power user, casual user, lapsed user)
How much variance reduction to expect: The variance reduction is proportional to how much of the total metric variance is explained by the strata. If user tenure explains 30% of the variance in your metric, stratifying on tenure can reduce your required sample size by up to 30%.
Post-stratification analysis: If you did not stratify at assignment time, you can still apply post-stratification adjustment at analysis time — weighting each stratum's treatment effect by its true population proportion. This is less efficient than pre-stratification but still reduces variance.
import numpy as np
from scipy.stats import ttest_ind
# Simulate stratified assignment
np.random.seed(42)
strata_sizes = {'new': 1000, 'returning': 2000, 'power': 500}
treatment, control = [], []
for stratum, n in strata_sizes.items():
# Assign half of each stratum to each group
base_metric = {'new': 5, 'returning': 10, 'power': 25}[stratum]
t_samples = np.random.normal(base_metric + 0.5, 3, n // 2)
c_samples = np.random.normal(base_metric, 3, n // 2)
treatment.extend(t_samples)
control.extend(c_samples)
t_stat, p_val = ttest_ind(treatment, control)
print(f"Stratified test: t={t_stat:.3f}, p={p_val:.4f}")
Q12: How do you handle a situation where multiple metrics in one experiment give conflicting results?¶
Show answer
Conflicting metrics are the norm, not the exception, in A/B tests. The key is to have a decision framework in place before you see the results — not improvised after.
Common conflict patterns:
-
Primary metric improves, but a guardrail metric worsens: the treatment is not acceptable as-is. Stop. The improvement in the primary metric is not worth the harm. Iterate on the design.
-
Primary metric is flat, but a secondary metric improves: informative but not a "ship" signal on its own. Use the secondary metric improvement as a hypothesis for the next experiment — can you design a change that moves both?
-
Primary metric improves in one segment but declines in another: a heterogeneous treatment effect. Understand the mechanism. Is the decline segment large enough to make the overall effect net-negative? Is there a way to apply the treatment selectively?
-
Short-term primary metric improves, but a long-term leading indicator worsens: classic recommendation / engagement tension. The change drives short-term clicks but reduces diversity or satisfaction signals that predict long-term retention. Usually a strong signal not to ship.
Pre-specifying your decision hierarchy: Before the experiment: 1. Define the primary metric and what would constitute "ship" (e.g., primary metric lifts by at least MDE, no guardrail metric significantly worsens) 2. Define secondary metrics as informational — they do not override the primary metric decision 3. Define guardrail metrics as hard constraints — a significant violation stops the ship decision
The mistake to avoid: treating the experiment as a negotiation where you weigh a dozen metrics and find a subjective winner. This is p-hacking with extra steps. Your decision framework should be specifiable before any results come in.
Q13: What is a holdout test, and how is it different from running a standard A/B test?¶
Show answer
Holdout tests (also called long-term holdouts or holdback experiments) are specifically designed to measure the sustained, long-term effect of a launched feature — not the short-term effect during an experiment window.
How a holdout test works: 1. Launch the feature to 95–99% of users 2. Hold back 1–5% of users on the old experience 3. Monitor the metric delta between holdout and general population for 30–90+ days post-launch
How it differs from a standard A/B test: - An A/B test runs for 1–4 weeks during development, measures the short-term effect, and terminates at a decision point - A holdout test runs after the launch decision, measuring the long-term effect with most users on the new experience
What holdout tests reveal: - Whether the short-term A/B test lift was driven by novelty (the holdout will show the gap closing over time) - Network effects that only manifest when most of the network has adopted the feature - Long-term changes in user habits (some features change how users use the product — this takes weeks to stabilise)
Practical challenge: the holdout group must be carefully maintained. They cannot receive the feature through any path — not through a/b test assignment, not through feature flags, not through seeing content created by feature users that implies access to the feature.
Q14: What is an A/A test and when do you run one?¶
Show answer
An A/A test (also called a null experiment) assigns users to two groups where both groups receive the same, unchanged experience. No treatment is applied.
Why you run A/A tests:
1. Validate your experimentation infrastructure If your experiment framework is working correctly, an A/A test should show no statistically significant difference between the two groups on your primary metric. If it does show a significant difference, your randomisation is broken, your logging is incorrect, or your statistical analysis has a bug.
2. Calibrate false positive rates Run many A/A tests. The fraction that produce p < 0.05 should be approximately 5%. If it is higher, you have a systematic issue (correlated observations, incorrect variance estimation, biased randomisation).
3. Measure metric variance An A/A test gives you a clean estimate of your metric's natural variance — useful for sample size calculations that require variance as an input.
4. Build organisational trust Before stakeholders trust your A/B test results, showing that your system produces null results when there is truly no difference builds credibility.
When to run A/A tests: - When setting up a new experiment platform or a new metric for the first time - When onboarding a new team to the experiment framework - When something unexpected happened in a recent experiment that you cannot explain - Periodically as a health check on the infrastructure
Common issue A/A tests catch: Session-ID vs User-ID hashing causing uneven splits; logging events that are double-counted; metrics computed with incorrect denominators.
Q15: What is geo experimentation, and when would you choose it over user-level randomisation?¶
Show answer
Geo experimentation randomises at the geographic level — entire cities, DMA regions, countries, or postal codes are assigned to treatment or control — rather than at the individual user level.
When to choose geo experiments:
- Marketplace interference: in a two-sided marketplace (Uber, Instacart, DoorDash), driver-side and rider-side randomisation cannot both be independent. Assigning entire cities to treatment avoids cross-contamination within a market.
- Offline / physical world interventions: price changes, promotional campaigns, out-of-home advertising, sales team deployments cannot be randomised by user.
- Brand and awareness campaigns: TV, podcast, or influencer campaigns affect all users in a market simultaneously — you cannot randomise by user.
- Features that require market-level critical mass: social features, local discovery, community features only work when enough users in a geographic area have them.
Challenges of geo experiments:
- Few randomisation units: if you have 20 cities, you have 20 randomisation units (not 2M users). Statistical power is much lower — you need a large per-city effect to detect anything reliably.
- Market-level heterogeneity: cities differ on baseline metrics, demographics, seasonality. Requires careful matching of treatment and control markets on pre-experiment trends.
- Interference between markets: if people travel between cities or the product is not geographically contained, contamination can occur.
- Analysis complexity: requires difference-in-differences or synthetic control analysis, not a simple t-test.
Best practice: pair geo experiments with pre-experiment balance checks (are treatment and control cities similar on key metrics before the treatment?), and confirm parallel pre-period trends before interpreting results.