Experiment Design Guide

How to design a rigorous A/B test that produces trustworthy results.

Pre-Experiment Checklist

1. Define the hypothesis clearly

• Bad: "The new design is better"
• Good: "Reducing signup form fields from 8 to 4 increases completion rate by at least 5pp"

2. Choose the primary metric

• Must be directly affected by the change
• Must be measurable within the experiment timeframe
• Must have low enough variance to detect your MDE

3. Calculate sample size

• Use required_sample_size_proportions() for conversion metrics
• Use required_sample_size_means() for continuous metrics
• Rule of thumb: if you can't run for at least 7 days, your MDE is too small

4. Define guardrail metrics

• Revenue, error rates, engagement — things that must not degrade
• Set specific thresholds for each

5. Plan for multiple testing

• One primary metric (p<0.05 threshold)
• Apply Bonferroni correction for secondary metrics

Common Pitfalls

Sample Size Rules of Thumb

These assume alpha=0.05, power=0.80, two-sided test.

Interpreting A/B Test Results

How to read your test results and make the right decision.

Decision Framework

                    p < 0.05?
                   /         \
                 YES          NO
                /               \
     Lift > 0?              Underpowered?
     /      \               /         \
   YES      NO            YES          NO
    |        |             |            |
  SHIP     KILL        EXTEND      ACCEPT NULL

Reading the Numbers

P-value

• What it is: Probability of seeing this result (or more extreme) if there's truly no effect
• What it's NOT: Probability that the hypothesis is wrong
• Threshold: p < 0.05 is conventional, but not magic. p=0.049 and p=0.051 are practically identical.

Confidence Interval

• What it is: Range of plausible true effect sizes given your data
• If CI excludes zero: Statistically significant
• Width indicates precision: Wide CI = uncertain estimate; narrow = precise

Effect Size

• Absolute lift: Percentage point difference (e.g., +2pp)
• Relative lift: Percent change from baseline (e.g., +20%)
• When to use which: Absolute for understanding magnitude; relative for communicating impact

Bayesian Probability

• P(B>A): Probability that treatment is truly better than control
• Expected loss: How much you'd "lose" by choosing wrong
• Threshold: P(B>A) > 0.95 is roughly analogous to p < 0.05

Common Scenarios and What to Do

Scenario 1: Significant positive result

• Action: Ship if guardrails pass
• Caveat: Check that the effect is practically meaningful, not just statistically significant

Scenario 2: Significant negative result

• Action: Kill the experiment
• Learning: Document why the change hurt; inform future iterations

Scenario 3: Not significant, positive direction

• Options:
• Extend the experiment (if underpowered)
• Accept that the effect is smaller than your MDE
• Consider whether a smaller true effect would still be worth shipping

Scenario 4: Not significant, near zero

• Action: Accept the null. The change has no meaningful effect.
• Learning: The hypothesis was wrong. Move on to the next idea.

Guardrail Interpretation

Even if the primary metric wins, check ALL guardrails:

• Revenue degradation → Don't ship (you're trading money for signups)
• Error rate increase → Don't ship (technical quality issue)
• Engagement drop → Investigate (might be acceptable tradeoff)

Communicating Results to Stakeholders

Template:

We tested [change] vs. the current experience over [duration] with [N] users.

The primary metric ([metric name]) [improved/did not change/degraded] by [lift]

(p=[value], 95% CI: [range]).

Recommendation: [Ship / Don't ship / Extend].

Next step: [specific action].