← Back to all products

Experiment Tracking Pack

$29

Experiment tracking with W&B and MLflow, custom dashboards, metrics comparison, and reproducibility tooling.

📁 18 files🏷 v1.0.0
JSONMarkdownPythonYAMLMLflow

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 18 files

experiment-tracking-pack/ ├── LICENSE ├── README.md ├── config.example.yaml ├── examples/ │ ├── compare_experiments.py │ └── track_training_run.py ├── guides/ │ ├── choosing-a-tracker.md │ └── experiment-tracking-guide.md ├── src/ │ └── exp_tracking/ │ ├── __init__.py │ ├── base.py │ ├── compare.py │ ├── custom_tracker.py │ ├── factory.py │ ├── mlflow_tracker.py │ ├── sweep.py │ └── wandb_tracker.py └── tests/ ├── test_base.py └── test_custom_tracker.py

📖 Documentation Preview README excerpt

Experiment Tracking Pack

One experiment-tracking interface, three interchangeable backends. Instrument

your ML training loop once and send the results to Weights & Biases,

MLflow, or a zero-dependency local SQLite store — switching between them

with a single line of configuration.

The W&B and MLflow SDKs are imported lazily, so import exp_tracking (and the

entire test suite) works with nothing but the Python standard library

installed. You only need wandb or mlflow when you actually use that backend.

Features

  • Unified ExperimentTracker interface — start_run, log_params,

log_metrics, log_artifact, end_run, plus list_runs / get_metric_history

for reading results back.

  • Three backends, identical API:
  • CustomTracker — local, SQLite-backed, pure stdlib (great for tests/CI/offline).
  • WandbTracker — Weights & Biases, with artifact and sweep support.
  • MlflowTracker — MLflow Tracking, local mlruns/ or a self-hosted server.
  • run() context manager that always finalizes a run — FINISHED on success,

FAILED (and re-raised) on exception.

  • Hyperparameter sweeps — grid and random search expansion, plus

build_wandb_sweep_config for W&B Bayesian sweeps.

  • Cross-backend comparison — compare_runs builds a table from any backend,

rendered as ASCII, Markdown, or CSV, with best_run to pick the winner.

  • Config-driven backend selection — choose the backend from a YAML/JSON file.
  • Metric validation — non-numeric / NaN / inf values are rejected with a

clear local error before they reach a backend.

Installation

The pack itself needs no installation — drop src/exp_tracking/ onto your

PYTHONPATH (the examples and tests do this automatically). Install a backend

SDK only if you use it:


pip install wandb        # for the W&B backend  (then: wandb login)
pip install mlflow       # for the MLflow backend
pip install pyyaml       # only if you load a .yaml config (JSON needs nothing)
# the local SQLite backend, sweeps, and comparison need no third-party packages

Quick Start


from exp_tracking import get_tracker, compare_runs, format_table

# Pick a backend by name. "custom" runs locally with zero setup.
tracker = get_tracker("custom", db_path="runs.db", project="image-classification")

# Track a training loop. The context manager finalizes the run for you.
with tracker.run(run_name="adam-lr1e-3", config={"lr": 1e-3, "epochs": 10}) as run:
    for epoch in range(10):
        loss = 1.0 / (epoch + 1)
        tracker.log_metrics({"train_loss": loss, "val_accuracy": 0.5 + epoch / 20},
                            step=epoch)

# Compare every run in the project and print a table.
table = compare_runs(tracker, metric_keys=["val_accuracy", "train_loss"])

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/exp_tracking/base.py """ Abstract experiment-tracking interface and shared data models. Every backend in this pack — Weights & Biases (:mod:`exp_tracking.wandb_tracker`), MLflow (:mod:`exp_tracking.mlflow_tracker`), and the bundled SQLite-backed local tracker (:mod:`exp_tracking.custom_tracker`) — implements the same :class:`ExperimentTracker` interface defined here. Code written against this interface can switch backends by changing a single line; see :mod:`exp_tracking.factory`. This module is intentionally pure standard library so it imports cleanly in any environment, including CI runners that have neither ``wandb`` nor ``mlflow`` installed. Backend-specific imports live inside the concrete tracker modules and are loaded lazily, only when that backend is actually used. """ from __future__ import annotations import abc import enum from collections.abc import Iterator, Mapping from contextlib import contextmanager from dataclasses import dataclass, field from numbers import Real from typing import Any, Optional __all__ = [ "TrackerError", "RunStatus", "RunInfo", "MetricPoint", "ExperimentTracker", "flatten_params", "coerce_metric_value", ] class TrackerError(RuntimeError): """Raised when a tracker is used incorrectly. Examples include logging metrics before :meth:`ExperimentTracker.start_run` has been called, or passing a non-numeric value where a metric is expected. Using a dedicated exception type lets callers distinguish *their* misuse from genuine backend/network failures raised by ``wandb`` or ``mlflow``. """ class RunStatus(enum.Enum): """Lifecycle state of a single experiment run. # ... 276 more lines ...
Buy Now — $29 Back to Products