A/B Testing Statistical Framework

$39

Complete A/B testing toolkit: sample size calculator, statistical significance tests, Bayesian analysis, and results reporting templates.

📁 34 files

YAMLMarkdownPython

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 34 files

ab-testing-framework/ ├── LICENSE ├── README.md ├── configs/ │ └── experiment_config.yaml ├── free-sample.zip ├── guide/ │ ├── experiment_design.md │ └── interpreting_results.md ├── guides/ │ ├── experiment_design.md │ └── interpreting_results.md ├── index.html ├── requirements.txt ├── src/ │ └── abtest/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── bayesian.cpython-312.pyc │ │ ├── cuped.cpython-312.pyc │ │ ├── frequentist.cpython-312.pyc │ │ ├── main.cpython-312.pyc │ │ ├── reporter.cpython-312.pyc │ │ ├── sample_size.cpython-312.pyc │ │ └── sequential.cpython-312.pyc │ ├── bayesian.py │ ├── cuped.py │ ├── frequentist.py │ ├── main.py │ ├── reporter.py │ ├── sample_size.py │ └── sequential.py └── tests/ ├── __init__.py ├── __pycache__/ │ ├── __init__.cpython-312.pyc │ ├── test_bayesian.cpython-312.pyc │ ├── test_frequentist.cpython-312.pyc │ └── test_sample_size.cpython-312.pyc ├── test_bayesian.py ├── test_frequentist.py └── test_sample_size.py

📖 Documentation Preview README excerpt

A/B Testing Statistical Framework

A complete A/B testing toolkit implementing real statistical methods from scratch. Includes sample size calculation, frequentist tests, sequential testing, Bayesian analysis, and CUPED variance reduction — all runnable with Python's standard library + scipy.

Features

Sample Size Calculator — Power analysis for proportions and continuous metrics
Frequentist Tests — Z-test for proportions, Welch's t-test, chi-square goodness-of-fit
Sequential Testing — O'Brien-Fleming spending function for early stopping
Bayesian A/B — Beta-Binomial model with posterior probability of winning
CUPED Variance Reduction — Pre-experiment covariate adjustment to reduce required sample size
Results Reporter — Generate structured experiment summaries with confidence intervals

Quick Start


pip install -r requirements.txt
python -m src.abtest.main

Structure


src/abtest/
├── __init__.py           # Package exports
├── main.py               # Demo runner
├── sample_size.py        # Power analysis and sample size calculation
├── frequentist.py        # Z-test, t-test, chi-square implementations
├── sequential.py         # Group sequential testing with alpha spending
├── bayesian.py           # Beta-Binomial Bayesian A/B testing
├── cuped.py              # CUPED variance reduction
├── reporter.py           # Structured results reporting
tests/
├── test_sample_size.py   # Validates power calculations against known values
├── test_frequentist.py   # Validates test statistics
├── test_bayesian.py      # Validates posterior calculations
configs/
├── experiment_config.yaml  # Example experiment configuration
guides/
├── experiment_design.md    # How to design a rigorous experiment
├── interpreting_results.md # How to read and act on test results

Usage Examples

Calculate Required Sample Size


from src.abtest.sample_size import required_sample_size_proportions

n = required_sample_size_proportions(
    baseline_rate=0.12,          # Current conversion rate: 12%
    minimum_detectable_effect=0.02,  # Want to detect 2pp lift (to 14%)
    alpha=0.05,                  # 5% significance level
    power=0.80,                  # 80% power
)
print(f"Need {n:,} users per variant")

Run a Frequentist Test

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/abtest/bayesian.py """ Bayesian A/B testing using the Beta-Binomial conjugate model. Computes posterior distributions for conversion rates and the probability that one variant beats another. No MCMC needed — the Beta-Binomial model has a closed-form posterior. """ from __future__ import annotations import math from dataclasses import dataclass @dataclass class BayesianResult: """Result of a Bayesian A/B test.""" control_posterior_alpha: float control_posterior_beta: float treatment_posterior_alpha: float treatment_posterior_beta: float control_posterior_mean: float treatment_posterior_mean: float prob_treatment_wins: float expected_lift: float credible_interval_lift: tuple[float, float] risk_of_choosing_treatment: float interpretation: str def bayesian_ab_test( control_successes: int, control_total: int, treatment_successes: int, treatment_total: int, prior_alpha: float = 1.0, prior_beta: float = 1.0, num_simulations: int = 100_000, ) -> BayesianResult: """Run a Bayesian A/B test using Beta-Binomial model. The Beta-Binomial is the conjugate prior for binomial data: - Prior: Beta(alpha, beta) — default is uniform Beta(1,1) - Posterior: Beta(alpha + successes, beta + failures) The probability that treatment > control is computed via Monte Carlo sampling from both posteriors. Args: control_successes: Conversions in control. # ... 149 more lines ...

Buy Now — $39 Back to Products