Choosing a Tracker: W&B vs MLflow vs Local SQLite

This pack ships three backends behind one interface so you are never locked in.

But which should you actually *use*? This guide gives you a decision framework,

an honest comparison, and migration notes. Because every backend implements the

same ExperimentTracker interface, switching later is a one-line change — so

optimize for where you are today, not where you might be in two years.

TL;DR decision tree

Are you offline, in CI, or writing tests?
└─ Yes → custom (local SQLite). Zero setup, zero network, fully deterministic.
└─ No → Do you need a hosted UI, sweeps, and team sharing with no ops work?
        └─ Yes → wandb. Best-in-class dashboards and hyperparameter sweeps.
        └─ No → Do you want self-hosted, open-source, and model-registry features
                under your own control?
                └─ Yes → mlflow. Run your own server, own your data.
                └─ Unsure → start with custom, switch when a real need appears.

Side-by-side comparison

When to pick the local SQLite backend

The custom tracker is not a toy — it is the right default in several real

situations:

• Unit tests and CI. It needs no credentials and no network, so test suites

stay fast and hermetic. This pack's own tests use it.

• Air-gapped or regulated environments where data cannot leave the machine.
• Quick local experiments where spinning up a server is more work than the

experiment itself.

• Teaching and demos where you want the audience to run code immediately.

Its limits are equally clear: there is no hosted UI (you compare via

format_table/to_markdown), and "sharing" means sharing the SQLite file. When

you outgrow it, the interface makes moving up painless.

When to pick Weights & Biases

Choose wandb when you want a polished, hosted experience with essentially no

operational burden:

• Rich dashboards with interactive plots, parallel-coordinates views, and

run tables your whole team can browse.

• Hyperparameter sweeps with Bayesian optimization out of the box — build the

config with build_wandb_sweep_config and launch agents.

• Team workflows: reports, run sharing, and commenting are first-class.

Trade-offs: it is a third-party SaaS (your run data lives on their servers,

though an offline mode exists), and seats cost money beyond the free tier. Use

mode="offline" on locked-down machines and wandb sync later.

When to pick MLflow

Choose mlflow when you want open-source tracking you control end to end:

• Self-hosting: point the tracker at your own server

(tracking_uri="http://mlflow.internal:5000") or even just a local mlruns/

directory with no server at all.

• Model registry: MLflow's registry is excellent for staging/production model

promotion workflows.

• No per-seat cost: you pay for infrastructure, not licenses.

Trade-offs: someone has to run and maintain the server for team use, and the UI,

while solid, is less flashy than W&B's. For many organizations that control is

exactly the point.

Cost and hosting at a glance

• Local SQLite: free; the only "cost" is that collaboration is manual.
• W&B: generous free tier for individuals; paid plans for teams and private

hosting. You trade dollars for zero ops.

• MLflow: the software is free; you pay in infrastructure and maintenance

time for a shared server (a small VM is often enough).

Migrating between backends

Because all three implement ExperimentTracker, your *training code* never

changes — only how you construct the tracker:

# Local during development...
tracker = get_tracker("custom", db_path="runs.db", project="demo")

# ...W&B when you want dashboards...
tracker = get_tracker("wandb", project="demo", entity="your-org")

# ...MLflow when you self-host.
tracker = get_tracker("mlflow", experiment_name="demo",
                      tracking_uri="http://mlflow.internal:5000")

Better still, select the backend from config so it's an ops decision, not a code

change:

from exp_tracking import load_config, tracker_from_config

tracker = tracker_from_config(load_config("config.yaml"))

There is no automated cross-backend *history* migration — runs already recorded

in W&B don't move to MLflow. But because comparison reads through the same

interface, you can point compare_runs at whichever backend holds the runs you

care about. In practice teams run custom in CI and a hosted backend for real

training, which this design supports natively.

A reasonable default path

1. Start on custom while you prototype — instant, offline, testable.

2. Graduate to wandb when you want dashboards and sweeps without ops, or to

self-hosted mlflow when data control and a model registry matter more.

3. Keep custom in your test suite forever — it is the fastest, most

reliable way to verify your tracking code without external dependencies.

You are not choosing a tracker for life. You are choosing the one that removes

the most friction from your *next* experiment.

The Experiment Tracking Guide

A practical guide to instrumenting machine-learning experiments so that, six

months from now, you (or a teammate) can answer the only questions that matter:

*which run was best, what made it best, and can I reproduce it?* The patterns

below are backend-neutral — they apply whether you log to Weights & Biases,

MLflow, or the local SQLite tracker that ships with this pack.

Why track experiments at all

Untracked experiments rot. The notebook that produced your best F1 score gets

edited, the hyperparameters live in a Slack message, and the checkpoint sits in a

folder named final_v3_REAL. Experiment tracking replaces that entropy with a

durable record: every run captures its configuration, its **metrics over

time, and its artifacts** (checkpoints, plots, datasets), keyed by a stable

run id. Three habits pay for themselves almost immediately:

1. Comparability — runs become rows in a table you can sort and filter.

2. Reproducibility — a run carries enough context to be re-created.

3. Collaboration — "look at run a1b2c3" beats "trust me, it worked."

The anatomy of a run

This pack models a run with three distinct kinds of data. Keeping them separate

is the single most useful convention in experiment tracking.

A param answers "how was this configured?" and never changes within a run. A

metric answers "how did it do over time?" and is recorded at every step. If you

find yourself logging a value that changes during training as a *param*, it

belongs in metrics — and vice versa.

Starting and ending runs cleanly

Always finalize a run, even when training crashes. The run() context manager

does this for you: a clean exit marks the run FINISHED; an exception marks it

FAILED and re-raises. That single guarantee keeps your dashboard honest — a

half-finished run is never silently left as "running" forever.

from exp_tracking import get_tracker

tracker = get_tracker("custom", db_path="runs.db", project="image-classification")

with tracker.run(run_name="resnet18-baseline", config=config, tags=["baseline"]):
    for epoch in range(epochs):
        train_loss = train_one_epoch(...)
        val_loss, val_acc = validate(...)
        tracker.log_metrics(
            {"train_loss": train_loss, "val_loss": val_loss, "val_accuracy": val_acc},
            step=epoch,
        )

If you prefer manual control, call start_run() / end_run(status) yourself —

just remember to handle the failure path.

Understanding step

The step argument is the x-axis of every metric plot. Pick one meaning for

it per project and stick to it — usually the epoch index or the global batch

counter. Consistency matters more than the choice: if half your runs step by

epoch and half by batch, their curves are not comparable. When you omit step,

this pack assigns a monotonically increasing integer, which is fine for simple

loops but explicit steps are strongly recommended for anything you'll compare.

Log training and validation metrics at the same step so they line up on a

shared axis. Logging train_loss every batch but val_loss every epoch makes

overlay plots misleading.

Naming and tagging conventions

Names and tags are how you find a run later. A little discipline goes a long way:

• Run names should encode the *idea*, not a timestamp: adam-lr1e-3-aug beats

run_2024_05_01_173455. The backend already stores creation time.

• Tags are for cross-cutting filters: baseline, ablation, gpu-a100,

dataset-v2. Tags let you pull "every ablation on dataset-v2" in one query.

• Projects group runs that are meaningfully comparable. A new dataset or a

fundamentally different task deserves a new project.

Reproducibility: capture enough to re-run

A run is reproducible when its record contains everything needed to recreate it.

At minimum, log these as params at the start of every run:

import subprocess, sys

config = {
    "seed": 42,
    "git_sha": subprocess.run(
        ["git", "rev-parse", "HEAD"], capture_output=True, text=True
    ).stdout.strip(),
    "python": sys.version.split()[0],
    "dataset_version": "v2",
    # ...plus all model/optimizer hyperparameters
}

Set and record your random seeds (random, numpy, your framework). Capturing

the git SHA means you can always check out the exact code that produced a result.

Logging the dataset version prevents the classic "the data changed under me"

mystery.

Artifacts and versioning

Treat artifacts as first-class outputs, not afterthoughts:

• Checkpoints: log the best model, not every epoch — storage adds up fast. A

common pattern is to log a single best.pt and overwrite it when validation

improves.

• Plots and reports: a confusion matrix or a calibration curve attached to

the run is worth a thousand scalar metrics during review.

• Inputs: for small datasets or splits, logging the exact data used closes

the reproducibility loop entirely.

The local backend copies artifacts into an artifacts// directory so a

run is self-contained and portable.

Comparing runs

Logging is only half the job; the payoff is comparison. Use compare_runs to

build a table across *any* backend, then rank it:

from exp_tracking import compare_runs, format_table, best_run

table = compare_runs(tracker, metric_keys=["val_accuracy", "val_loss"],
                     reduction="best_max")
print(format_table(table))
winner = best_run(table, "val_accuracy", mode="max")

By default the comparison shows only the params that differ across runs, so

the signal (what you changed) is not buried under the constants (what you didn't).

Choose a reduction that matches the question: "last" for "where did it end

up", "best_max"/"best_min" for "what's the best it ever reached".

Sweeps

When you move from one run to many, describe the search space declaratively and

let exp_tracking.sweep expand it. Grid search is exhaustive and reproducible;

random search covers high-dimensional spaces more efficiently per trial.

from exp_tracking import grid_search, build_wandb_sweep_config

trials = grid_search({"lr": [1e-2, 1e-3, 1e-4], "batch_size": [32, 64]})  # 6 configs

# Or hand the search space to a W&B Bayesian sweep:
sweep_config = build_wandb_sweep_config(
    space={"lr": (1e-5, 1e-1), "dropout": (0.0, 0.5), "optimizer": ["adam", "sgd"]},
    metric_name="val_accuracy", goal="maximize", method="bayes",
)

Common pitfalls

• Logging booleans as metrics. coerce_metric_value rejects raw True/False

on purpose — cast to 1.0/0.0 so your intent is explicit.

• Inconsistent step semantics across runs in the same project (see above).
• Forgetting to finalize a run after a crash — use the context manager.
• Tracking nothing reproducible. Metrics without the config, seed, and code

version are a leaderboard you can't act on.

• One giant project. If runs in a project aren't comparable, split it.

Track deliberately and your experiment history becomes an asset you mine, not a

graveyard you avoid. The next guide, choosing-a-tracker.md, helps you pick the

backend that fits your team.