MLflow Setup Guide

This guide takes you from "nothing installed" to a working MLflow tracking

server with a Postgres metadata store and an S3-compatible artifact store, then

shows how to point the kit at it. Read it once end-to-end; afterwards the

Model Registry Workflow guide covers promotion.

1. The three pieces of an MLflow deployment

Every MLflow setup is made of three independent pieces. Getting them straight

saves hours of confusion:

A common beginner setup collapses all three into the local filesystem

(file:./mlruns). That is fine for a laptop spike but it cannot be shared, has

no model registry over HTTP, and corrupts easily under concurrent writes. The

bundled docker/docker-compose.yml gives you the real, shareable topology with

one command.

2. Fastest path: the bundled Docker stack

# From the product root
docker compose -f docker/docker-compose.yml up -d

# Watch the tracking server come up (first boot pip-installs deps)
docker compose -f docker/docker-compose.yml logs -f mlflow

When the logs show Listening at: http://0.0.0.0:5000, open:

• MLflow UI -> http://localhost:5000
• MinIO console -> http://localhost:9001 (login minioadmin / minioadmin)

The stack wires everything together for you:

• Postgres is the --backend-store-uri.
• MinIO is the --artifacts-destination (bucket s3://mlflow).
• The server runs with --serve-artifacts, so **clients never need MinIO

credentials** -- they upload/download artifacts *through* the tracking server

using the mlflow-artifacts:/ scheme. This is the recommended modern pattern

and the reason your training code only needs one URL.

Tear it down (keeping data in named volumes) with:

docker compose -f docker/docker-compose.yml down
# add -v to also delete the postgres_data / minio_data volumes

3. Pointing the kit at the server

The kit centralises configuration in mlflow_starter.config.MLflowConfig. The

idiomatic flow is "read environment, then apply":

from mlflow_starter import MLflowConfig, configure

config = MLflowConfig.from_env()      # reads MLFLOW_TRACKING_URI, tags, etc.
config.experiment_name = "churn-modeling"
configure(config)                     # exports env + sets the MLflow client URIs

Set the environment once in your shell (or a .env consumed by your process

manager):

export MLFLOW_TRACKING_URI=http://localhost:5000
export MLFLOW_EXPERIMENT_NAME=churn-modeling
export MLFLOW_DEFAULT_TAGS='{"team": "ml-platform", "project": "churn"}'

MLFLOW_DEFAULT_TAGS is a JSON object string; the kit parses it and applies

those tags to every run opened through experiment_run.

Server-less fallback

No server running? Use a local file store. The example scripts already do this

automatically, but you can force it explicitly:

config = MLflowConfig(tracking_uri="file:./mlruns", experiment_name="quick-test")
configure(config)

Everything in tracking.py works against a file store except the *registry*

features, which require a database-backed server (Postgres here). That is the

single biggest reason to run the Docker stack.

4. Using a real cloud artifact store

MinIO speaks the S3 API, so moving to AWS S3, GCS (via the S3 interop), or any

S3-compatible object store is a configuration change, not a code change. For

real AWS S3 you simply drop the custom endpoint:

config = MLflowConfig(
    tracking_uri="https://mlflow.internal.example.com",
    artifact_location="s3://acme-ml-artifacts/mlflow/",
    s3_endpoint_url=None,          # None => default AWS endpoints
)
config.apply()

MLflowConfig.apply() only exports credentials that are actually set, so it

will never overwrite real IAM-role credentials with empty values. On AWS,

prefer an instance/role profile over static keys and leave

aws_access_key_id / aws_secret_access_key unset.

5. Verifying the connection

A 20-second smoke test that proves the whole chain (server + DB + artifacts) is

healthy:

import mlflow
from mlflow_starter import MLflowConfig, configure, experiment_run, log_metrics

configure(MLflowConfig.from_env())
with experiment_run("smoke-test") as run:
    log_metrics({"answer": 42})
    mlflow.log_text("hello from the smoke test", "notes.txt")
    print("run id:", run.info.run_id)

If this run appears in the UI and notes.txt is downloadable from its

Artifacts tab, your backend store and artifact store are both wired correctly.

If the run shows but the artifact 404s, your artifact store is misconfigured

(see Troubleshooting).

6. Troubleshooting

Connection refused to localhost:5000. The server container is still

pip-installing on first boot. Watch docker compose logs -f mlflow and wait for

the Listening at line.

Runs appear but artifacts fail to upload / 404. You are almost certainly

mixing the proxied-artifact and direct-S3 patterns. With this kit's stack the

server runs --serve-artifacts, so clients must use the tracking URL only --

do not also set MLFLOW_S3_ENDPOINT_URL in your *client* environment, or

the client will try to reach MinIO directly at a hostname it cannot resolve.

psycopg2 / database errors. The Postgres container failed its healthcheck

before MLflow started. Run docker compose ps and confirm postgres is

healthy; if it is restarting, you likely have a stale postgres_data volume

from an older Postgres major version -- remove it with down -v.

Registry calls raise RestException: ... not supported. You are pointed at

a file: store. Stage transitions and the model registry require the

database-backed server. Switch MLFLOW_TRACKING_URI to http://localhost:5000.

Stale model in serving. Loading models://Production always resolves

the *current* Production version. If a server keeps an old model, it cached it

at startup -- restart the scoring container after a promotion, or use aliases

plus a redeploy (see the registry workflow guide).

7. Production hardening checklist

Before this leaves your laptop:

1. Replace minioadmin / mlflow passwords; move them into a secrets manager.

2. Put the tracking server behind TLS and an auth proxy (MLflow has no built-in

auth in the open-source server).

3. Pin every image tag (postgres:16.3, a dated minio release) instead of

latest for reproducible rebuilds.

4. Back up the Postgres database -- it is the source of truth for your entire

experiment and model history.

5. Bake the MLflow server dependencies into a custom image so restarts do not

re-run pip install.

Model Registry Workflow

The model registry is what turns MLflow from an experiment notebook into a

deployment system. This guide explains the lifecycle the kit implements, when to

use stages versus aliases, and how to wire promotion into CI. It assumes you

have a database-backed tracking server running (see the

MLflow Setup Guide).

1. The lifecycle

A model in the registry is a named container (customer-churn-classifier) that

holds numbered versions (1, 2, 3...). Each version moves through stages:

   register            promote            promote
  (new version)  ->   Staging    ->     Production
        |                |                   |
        |                v                   v
        +----------> Archived  <--------  Archived
                    (rejected)          (superseded)

• None -- a freshly created version, not yet promoted.
• Staging -- a candidate undergoing validation (shadow traffic, holdout

metrics, manual review).

• Production -- the version currently serving real predictions.
• Archived -- retired versions, kept for audit and rollback.

mlflow_starter.registry.ModelRegistry gives you one method per transition so

this lifecycle reads like prose in your code.

2. Registering a version from a run

You never register "a file". You register a model that was **logged inside a

run**, which preserves the link between the deployed artifact and the exact

code, params, and metrics that produced it.

from mlflow_starter import ModelRegistry

registry = ModelRegistry()
version = registry.register_from_run(
    run_id="3f1c...e9",            # the run that logged the model
    artifact_path="model",         # the artifact_path used in log_model(...)
    name="customer-churn-classifier",
    description="GradientBoosting candidate, week 24 retrain.",
)
print("registered version", version)

Under the hood this:

1. Calls ensure_registered_model (idempotent -- a "already exists" error is

swallowed, so you can call it on every retrain).

2. Builds the runs://model source URI.

3. Creates the version and waits until it reports READY before returning,

so you never promote a half-uploaded artifact.

3. Promote with automatic archiving

Promotion is a one-liner, and by default the previous occupant of the target

stage is archived for you -- exactly one Production version at a time:

registry.promote_to_staging("customer-churn-classifier", version)
# ...validate...
registry.promote_to_production("customer-churn-classifier", version)

promote_to_production passes archive_existing=True, so the model that was in

Production rolls to Archived in the same atomic call. That guarantee -- "there

is always exactly one Production version" -- is what lets your serving layer

trust models:/customer-churn-classifier/Production.

If you need the lower-level control, call transition_stage directly; it

validates the stage name against the canonical MLflow spelling and raises a

clear ValueError on a typo like "production" (MLflow stages are

case-sensitive).

4. A validation gate before Production

Never promote straight to Production. The pattern the kit encourages is

"register -> Staging -> *gate* -> Production", where the gate loads the Staging

model and checks it against a hard metric threshold:

from sklearn.metrics import roc_auc_score

PRODUCTION_AUC_GATE = 0.80

registry.promote_to_staging(name, version)
staged = registry.load_model(name, stage="Staging", flavor="sklearn")
auc = roc_auc_score(y_holdout, staged.predict_proba(x_holdout)[:, 1])

if auc >= PRODUCTION_AUC_GATE:
    registry.promote_to_production(name, version)
else:
    registry.archive(name, version)        # rejected candidate, kept for audit

This is exactly what examples/promote_model_example.py runs end-to-end. The

key idea: the *same* artifact you validated is the one promoted -- there is no

re-training or re-serialisation between the gate and Production.

5. Stages vs aliases

MLflow 2.9 introduced aliases (named pointers like champion,

challenger) as a more flexible successor to stages. They do not replace the

need for a promotion workflow; they change how serving *references* a version.

The kit supports both. After promoting, you can also pin an alias:

registry.promote_to_production(name, version)
registry.set_alias(name, "champion", version)

# Serving by alias:
model = registry.load_by_alias(name, "champion", flavor="sklearn")

A pragmatic default: use stages for the single-Production-model case (most

teams, most of the time) and reach for aliases when you genuinely run more

than one live model per name.

6. Loading the right model in serving

Your scoring service should reference a *stage or alias*, never a hard version

number -- that is what makes promotion a zero-redeploy operation conceptually:

# Universal flavor -- works for any logged model
model = registry.load_model("customer-churn-classifier", stage="Production")
predictions = model.predict(features)

load_model(..., flavor="sklearn") returns the native estimator (so you can

call predict_proba); the default flavor="pyfunc" returns the universal

wrapper used by mlflow models serve. Use pyfunc for serving parity and the

native flavor for offline analysis.

Note: the open-source MLflow scoring server resolves the model **once at

startup**. After a promotion you must restart (or roll) the serving

containers for them to pick up the new Production version. See the deployment

module for RealtimeServingConfig, which generates that serving setup.

7. Rollback

Because superseded versions are archived rather than deleted, rollback is just

another transition -- promote the known-good old version back to Production:

registry.promote_to_production("customer-churn-classifier", "7")  # last good one

The currently-bad Production version is archived automatically by the same call.

Keep a record of the last-known-good version number in your deployment system so

this is a single, fast command during an incident.

8. Wiring promotion into CI

A clean separation of duties keeps promotion auditable:

1. Training pipeline (scheduled): trains, logs metrics, calls

register_from_run, and promotes the new version to Staging only.

2. Validation job (automated + optional human approval): loads the Staging

model, runs the metric gate plus any fairness/latency checks, and on success

calls promote_to_production.

3. Deploy job: triggered by the Production transition (MLflow webhooks or a

registry poll), rebuilds/rolls the serving image with

mlflow models build-docker.

Tag each version as it moves (registry.client.set_model_version_tag) with the

git SHA, CI run URL, and validation metrics so every Production model is fully

traceable back to the commit that produced it. This audit trail is the whole

point of using a registry instead of copying model.pkl around.