← Back to all products
$29
A/B Test Framework
Statistical A/B testing framework with experiment setup, significance testing, and result analysis.
JSONMarkdownPython
📄 Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample📁 File Structure 10 files
ab-test-framework/
├── LICENSE
├── README.md
├── examples/
│ └── sample_experiment.json
├── free-sample.zip
├── guide/
│ ├── 01_features.md
│ ├── 02_quick-start.md
│ ├── 03_output.md
│ └── 04_faq.md
├── index.html
└── src/
└── ab_test_framework.py
📖 Documentation Preview README excerpt
A/B Test Framework
Statistical A/B testing framework: experiment setup, significance testing, sample size calculation, and result analysis. Know when a winner is real, not noise.
Features
- Z-test for proportions — Industry-standard two-proportion z-test for conversion rate comparisons
- Chi-squared test — Alternative significance test (2×2 contingency table)
- Sample size calculator — Know how many visitors you need before starting
- Confidence levels — 90%, 95%, 99% with correct z-score thresholds
- Simulation mode — Generate synthetic experiments to test your setup
- Relative & absolute lift — Both metrics reported for full context
- Pretty terminal output — Clear, formatted results with winner declaration
- JSON export — Machine-readable results for automation pipelines
Requirements
- Python 3.10+
- No external dependencies (stdlib only)
Quick Start
# Analyze experiment results from a JSON file
python src/ab_test_framework.py analyze --results examples/sample_experiment.json
# Analyze with 99% confidence level
python src/ab_test_framework.py analyze --results examples/sample_experiment.json --confidence 0.99
# Simulate an experiment (5000 visitors, 3.2% vs 4.1% conversion)
python src/ab_test_framework.py simulate --visitors 5000 --rate-a 0.032 --rate-b 0.041
# Calculate minimum sample size
python src/ab_test_framework.py sample-size --baseline 0.05 --mde 0.10
# Save results to JSON
python src/ab_test_framework.py analyze --results data.json --output report.json
Input Format
JSON
{
"control": {
"name": "Original Checkout",
"visitors": 5000,
"conversions": 160
},
"treatment": {
"name": "New Checkout",
"visitors": 5000,
"conversions": 205
}
}
CSV
... continues with setup instructions, usage examples, and more.
📄 Code Sample .py preview
src/ab_test_framework.py
#!/usr/bin/env python3
"""
A/B Test Framework — Analytics Hub (DataNest)
Statistical A/B testing framework: experiment setup, visitor assignment,
chi-squared and z-test significance analysis. Know when a winner is real,
not noise.
Usage:
python ab_test_framework.py --results experiment.json --analyze
python ab_test_framework.py --results experiment.csv --confidence 0.99
python ab_test_framework.py --simulate --visitors 5000 --rate-a 0.032 --rate-b 0.041
Dependencies: Python 3.10+ stdlib only (no pip packages)
License: MIT
"""
from __future__ import annotations
import argparse
import csv
import json
import logging
import math
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
logger = logging.getLogger("ab_test_framework")
# ---------------------------------------------------------------------------
# Constants — common significance thresholds
# ---------------------------------------------------------------------------
# Standard confidence levels and their z-scores for two-tailed tests.
# 90% is exploratory, 95% is the industry standard, 99% is conservative.
CONFIDENCE_Z: dict[float, float] = {
0.90: 1.645,
0.95: 1.960,
0.99: 2.576,
}
# ---------------------------------------------------------------------------
# Data models
# ---------------------------------------------------------------------------
@dataclass
class Variant:
"""One arm of an A/B experiment (control or treatment)."""
name: str
# ... 485 more lines ...