← Back to all products

Experiment Tracking Setup

$29

MLflow and Weights & Biases configurations with experiment comparison, model registry, and artifact management.

📁 31 files
MarkdownYAMLPythonMLflow

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 31 files

experiment-tracking-setup/ ├── LICENSE ├── README.md ├── configs/ │ ├── mlflow_config.yaml │ ├── tracking_config.yaml │ └── wandb_config.yaml ├── free-sample.zip ├── guide/ │ ├── 01-registry-workflow.md │ └── 02-setup-guide.md ├── guides/ │ ├── registry-workflow.md │ └── setup-guide.md ├── index.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── artifact_manager.cpython-312.pyc │ │ ├── experiment_logger.cpython-312.pyc │ │ ├── model_registry.cpython-312.pyc │ │ ├── run_comparison.cpython-312.pyc │ │ └── stdlib_tracker.cpython-312.pyc │ ├── artifact_manager.py │ ├── experiment_logger.py │ ├── model_registry.py │ ├── run_comparison.py │ └── stdlib_tracker.py └── tests/ ├── __init__.py ├── __pycache__/ │ ├── __init__.cpython-312.pyc │ ├── test_experiment_logger.cpython-312.pyc │ └── test_run_comparison.cpython-312.pyc ├── test_experiment_logger.py └── test_run_comparison.py

📖 Documentation Preview README excerpt

Experiment Tracking Setup

Unified experiment logging, run comparison, model registry, and artifact management — with a stdlib JSON fallback that runs locally with zero dependencies.

Switch between MLflow, Weights & Biases, and local JSON tracking by changing one config value. Your training code stays the same.

What You Get

  • Unified ExperimentLogger that wraps MLflow, W&B, or a local JSON tracker behind one API
  • Stdlib JSON tracker that works with zero external packages — runs in CI, on laptops, and in air-gapped environments
  • Run comparison and leaderboard generator with Markdown and CSV export
  • Hyperparameter sensitivity analysis to identify which params matter most
  • Model registry with version tracking, stage transitions (staging → production → archived), and audit trail
  • Artifact manager with content-addressed dedup and automatic file categorization
  • Metric buffering for high-frequency logging without overwhelming remote backends
  • Auto-detection of available backends — installs MLflow? It switches automatically
  • Environment capture — every run records git hash, Python version, hostname, and timestamp
  • MLflow and W&B configuration templates ready for team deployment

Quick Start


pip install -r requirements.txt   # optional — stdlib tracker needs nothing

python -c "
from src.experiment_logger import ExperimentLogger

exp = ExperimentLogger(backend='stdlib')

with exp.run('my-first-experiment', run_name='baseline') as run_id:
    exp.log_params({'model': 'random_forest', 'n_estimators': 100})
    for epoch in range(10):
        loss = 1.0 / (epoch + 1)
        exp.log_metric('loss', loss, step=epoch)
    exp.set_tag('status', 'baseline')

print(f'Run saved: {run_id}')
print('Check ./experiments/runs/ for the JSON output.')
"

Then generate a leaderboard:


from src.run_comparison import load_runs_from_directory, generate_leaderboard

runs = load_runs_from_directory("./experiments/runs")
lb = generate_leaderboard(runs, metric="loss", ascending=True)
print(lb.to_markdown())

Backend Switching


from src.experiment_logger import ExperimentLogger

# Local development — zero dependencies
exp = ExperimentLogger(backend="stdlib")

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/artifact_manager.py """ Artifact Manager — Organize, Version, and Retrieve ML Artifacts ================================================================ Handles the file-management side of experiment tracking: storing model checkpoints, datasets, figures, logs, and any other binary outputs tied to a training run. The core challenge this solves: training produces files everywhere (``/tmp/model.pkl``, ``./outputs/plot.png``, ``checkpoints/epoch_10.pt``), and after 50 runs you have no idea which file belongs to which run. This module gives every artifact a deterministic, content-addressed storage location tied to its run. Storage layout:: base_dir/ ├── artifacts/ │ ├── <run_id>/ │ │ ├── models/ │ │ │ ├── model.pkl │ │ │ └── model.onnx │ │ ├── metrics/ │ │ │ └── classification_report.json │ │ ├── figures/ │ │ │ └── confusion_matrix.png │ │ └── data/ │ │ └── test_predictions.csv │ └── ... └── manifest.json # Global artifact index Content addressing ~~~~~~~~~~~~~~~~~~ Each artifact is hashed (SHA-256) on ingest. If the same file is logged twice (e.g., a config that didn't change between runs), the manifest records the duplicate without wasting disk space — unless you explicitly disable dedup. """ from __future__ import annotations import hashlib import json import logging import os import shutil from dataclasses import asdict, dataclass, field from datetime import datetime, timezone from pathlib import Path from typing import Any, Dict, List, Optional # ... 401 more lines ...
Buy Now — $29 Back to Products