Experiment Tracking Setup

Experiment Tracking Setup

Unified experiment logging, run comparison, model registry, and artifact management — with a stdlib JSON fallback that runs locally with zero dependencies.

Switch between MLflow, Weights & Biases, and local JSON tracking by changing one config value. Your training code stays the same.

What You Get

• Unified ExperimentLogger that wraps MLflow, W&B, or a local JSON tracker behind one API
• Stdlib JSON tracker that works with zero external packages — runs in CI, on laptops, and in air-gapped environments
• Run comparison and leaderboard generator with Markdown and CSV export
• Hyperparameter sensitivity analysis to identify which params matter most
• Model registry with version tracking, stage transitions (staging → production → archived), and audit trail
• Artifact manager with content-addressed dedup and automatic file categorization
• Metric buffering for high-frequency logging without overwhelming remote backends
• Auto-detection of available backends — installs MLflow? It switches automatically
• Environment capture — every run records git hash, Python version, hostname, and timestamp
• MLflow and W&B configuration templates ready for team deployment

Quick Start

pip install -r requirements.txt   # optional — stdlib tracker needs nothing

python -c "
from src.experiment_logger import ExperimentLogger

exp = ExperimentLogger(backend='stdlib')

with exp.run('my-first-experiment', run_name='baseline') as run_id:
    exp.log_params({'model': 'random_forest', 'n_estimators': 100})
    for epoch in range(10):
        loss = 1.0 / (epoch + 1)
        exp.log_metric('loss', loss, step=epoch)
    exp.set_tag('status', 'baseline')

print(f'Run saved: {run_id}')
print('Check ./experiments/runs/ for the JSON output.')
"

Then generate a leaderboard:

from src.run_comparison import load_runs_from_directory, generate_leaderboard

runs = load_runs_from_directory("./experiments/runs")
lb = generate_leaderboard(runs, metric="loss", ascending=True)
print(lb.to_markdown())

Backend Switching

from src.experiment_logger import ExperimentLogger

# Local development — zero dependencies
exp = ExperimentLogger(backend="stdlib")

*... continues with setup instructions, usage examples, and more.*