← Back to all products
$29
A/B Testing Framework
A/B testing framework for landing pages with split traffic, statistical significance, and reporting.
JSONMarkdownPython
📄 Product Preview
Try the interactive reader and demo tools below, or get the full product with all content unlocked.
📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample📁 File Structure 9 files
ab-testing-framework/
├── LICENSE
├── README.md
├── examples/
│ └── experiment.json
├── free-sample.zip
├── guide/
│ ├── 01_features.md
│ ├── 02_cli-reference.md
│ └── 03_examples.md
├── index.html
└── src/
└── ab_testing_framework.py
📖 Documentation Preview README excerpt
A/B Testing Framework
Part of the Landing Lab by CodeVault
A self-contained A/B testing engine with deterministic variant assignment, conversion tracking, and statistical significance calculations using chi-squared and z-tests. No SaaS subscription needed.
Features
- Deterministic visitor-to-variant assignment (hash-based, no cookies needed)
- Conversion tracking with per-variant metrics
- Two-proportion z-test for pairwise comparisons
- Chi-squared test for multi-variant experiments
- Sample size calculator (know how long to run your test)
- CSV event log processing for batch analysis
- JSON and text report output
- Python stdlib only — zero pip dependencies
Quick Start
# Generate a report from an experiment config with data
python src/ab_testing_framework.py --config examples/experiment.json --report
# Assign a visitor to a variant
python src/ab_testing_framework.py --config examples/experiment.json --assign visitor_001
# Calculate required sample size
python src/ab_testing_framework.py --config examples/experiment.json --sample-size 0.05 0.20
# Load events from CSV and analyze
python src/ab_testing_framework.py --config examples/experiment.json --events events.csv --report
# JSON output for programmatic use
python src/ab_testing_framework.py --config examples/experiment.json --report --format json
CLI Reference
| Flag | Description |
|---|---|
--config, -c | Path to experiment JSON config (required) |
--events, -e | Path to events CSV file |
--assign VISITOR_ID | Assign a visitor and print the variant |
--convert VISITOR_ID | Record a conversion event |
--report | Generate experiment report |
--sample-size RATE EFFECT | Calculate required sample size |
--format, -f | Output format: text or json |
--verbose, -v | Enable verbose logging |
Configuration
See examples/experiment.json for a complete example.
Examples
Check if your test is statistically significant:
python src/ab_testing_framework.py --config examples/experiment.json --report
... continues with setup instructions, usage examples, and more.
📄 Code Sample .py preview
src/ab_testing_framework.py
#!/usr/bin/env python3
"""
A/B Testing Framework — Landing Lab by DataNest
A self-contained A/B testing engine that handles variant assignment,
conversion tracking, and statistical significance calculations using
chi-squared and z-tests. Feed it experiment configs and event logs,
and it tells you which variant is winning — with math to back it up.
Why this exists:
Most A/B testing tools are SaaS subscriptions that cost $50-200/month.
If you're an indie hacker running tests on a landing page, you just
need a script that assigns visitors to variants, logs conversions,
and runs the stats. This is that script.
Usage:
python ab_testing_framework.py --config experiment.json --report
python ab_testing_framework.py --config experiment.json --assign visitor123
python ab_testing_framework.py --config experiment.json --convert visitor123
python ab_testing_framework.py --events events.csv --analyze
License: MIT
"""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
import logging
import math
import sys
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
LOG = logging.getLogger("ab-testing-framework")
# Minimum sample size per variant before we consider results meaningful
MIN_SAMPLE_SIZE = 30
# Standard significance level (95% confidence)
DEFAULT_ALPHA = 0.05
# ... 589 more lines ...