DVC Setup Guide

A practical, end-to-end walkthrough for putting your datasets and ML pipeline

under version control with DVC (Data Version Control), and for using the

data_versioning toolkit in this product alongside it. Everything here uses

anonymized placeholders (your-org, s3://your-org-dvc-store); substitute your

own bucket and paths.

1. Why DVC (and what this toolkit adds)

Git is excellent for code and terrible for data: a 4 GB parquet file does not

belong in a git history. DVC solves this by storing a tiny pointer file

(*.dvc) in git while the real bytes live in a content-addressable cache that is

pushed to remote storage (S3, GCS, Azure Blob, SSH, or a shared filesystem).

The mental model:

The data_versioning package complements DVC in two places:

• Before you adopt DVC, or in environments where a full remote is overkill

(CI scratch, notebooks, ad-hoc audits), hashing.py + snapshots.py give you

the same content-addressable "did this dataset change?" check with zero

infrastructure.

• Alongside DVC, lineage.py, pipeline.py, and reproducibility.py let

you generate dvc.yaml programmatically, render the pipeline as a graph, and

capture the run environment that DVC does not record on its own.

2. Install

DVC is distributed on PyPI. Install the base package plus the extra for your

remote backend:

pip install "dvc[s3]"      # Amazon S3 / S3-compatible (MinIO, Cloudflare R2)
pip install "dvc[gs]"      # Google Cloud Storage
pip install "dvc[azure]"   # Azure Blob Storage
pip install "dvc[ssh]"     # any host you can reach over SSH

Verify the install and pin the major version in your project so teammates and CI

agree:

dvc --version          # expect 3.x
echo "dvc>=3,<4" >> requirements.txt

3. Initialize a repository

Run dvc init inside an existing git repository — DVC layers on top of git:

git init                 # if you have not already
dvc init
git add .dvc .dvcignore
git commit -m "chore: initialize DVC"

dvc init creates a .dvc/ directory (config + local cache) and a .dvcignore

file (same idea as .gitignore, but for DVC's own scans).

4. Configure a remote

The remote is where dataset bytes are pushed. Add one and mark it the default

with -d:

# S3 (or any S3-compatible endpoint)
dvc remote add -d storage s3://your-org-dvc-store/ml
dvc remote modify storage region eu-north-1

# Google Cloud Storage
dvc remote add -d storage gs://your-org-dvc-store/ml

# Azure Blob
dvc remote add -d storage azure://your-org-container/ml

# SSH / on-prem
dvc remote add -d storage ssh://user@storage.your-org.example.com/srv/dvc

Commit the remote config (it contains no secrets — credentials come from your

environment / cloud SDK):

git add .dvc/config
git commit -m "chore: configure default DVC remote"

Credentials never go in git. DVC reads AWS/GCP/Azure credentials from the

usual environment variables, ~/.aws/credentials, or instance metadata. Keep

secrets in your secret manager, exactly as the anonymization rules require.

5. Track a dataset

Hand a file or directory to dvc add. DVC moves the bytes into its cache,

writes a small *.dvc pointer, and updates .gitignore so the raw data is not

accidentally committed:

dvc add data/raw
git add data/raw.dvc data/.gitignore
git commit -m "data: track raw dataset with DVC"
dvc push                 # upload the bytes to the remote

The pointer file is what your teammates pull — dvc pull fetches the matching

bytes by hash.

Cross-check with this toolkit

Before and after dvc add, you can sanity-check the dataset's content hash with

the bundled hasher (no DVC call needed):

from data_versioning import hash_directory

print(hash_directory("data/raw").short_hash)   # stable, deterministic id

If that short hash matches what a teammate sees, you are looking at byte-identical

data — independent confirmation that dvc pull gave you the right version.

6. Define a pipeline (dvc.yaml)

A pipeline turns ad-hoc scripts into a reproducible DAG. See dvc/dvc.yaml in

this product for a complete four-stage example. The shape of one stage:

stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw
    params:
      - prepare.seed
      - prepare.test_size
    outs:
      - data/prepared

You can also generate dvc.yaml programmatically with pipeline.py, which

keeps stage definitions in code and avoids hand-editing YAML:

from data_versioning import Pipeline, Output

pipe = Pipeline()
pipe.add_stage(
    name="train",
    cmd="python src/train.py",
    deps=["src/train.py", "data/features"],
    params=["train.n_estimators", "train.max_depth"],
    outs=[Output("models/model.pkl", cache=True)],
    metrics=[Output("metrics.json", cache=False)],
)
pipe.write("dvc.yaml")          # emits valid YAML, no PyYAML dependency

7. Reproduce and inspect

dvc repro                 # run only the stages whose inputs changed
dvc dag                   # print the pipeline graph in your terminal
dvc metrics show          # show metrics from metrics.json
dvc params diff           # show which params changed vs the last commit
dvc metrics diff          # show how metrics moved vs the last commit

dvc repro is the heart of reproducibility: because every stage declares its

deps, params, and outs, DVC knows exactly what is stale and reruns the minimum.

8. The everyday workflow

# pull code + data to the exact versions recorded in this commit
git pull
dvc pull

# ... make changes (edit code, bump a param, refresh data) ...

dvc repro                 # rebuild what changed
git add dvc.lock metrics.json params.yaml
git commit -m "exp: tune max_depth=8"
dvc push                  # upload any new/changed artifacts
git push

dvc.lock is the auto-generated record of the exact hashes used in the last

successful run — always commit it. Together, git + dvc.lock + the remote let

anyone recreate your result byte-for-byte.

9. CI integration

A minimal CI job that verifies the pipeline still reproduces:

# .github/workflows/dvc.yml (illustrative)
steps:
  - uses: actions/checkout@v4
  - run: pip install "dvc[s3]" -r requirements.txt
  - run: dvc pull                       # fetch data by hash
  - run: dvc repro --dry                # fail if anything is stale/undeclared
  - run: dvc metrics diff --targets metrics.json

For pull requests, dvc metrics diff main posts a clean before/after table so

reviewers can see the effect of a change on model quality.

10. Troubleshooting

For the "changed every run" class of problems, hash the output directory with

data_versioning.hash_directory twice in a row: if the tree_hash differs, the

nondeterminism is in *your* code, not DVC. See reproducibility-guide.md for the

usual culprits and fixes.

Next steps

• Read reproducibility-guide.md for seeds, environment capture, and the

reproducibility checklist.

• Run python examples/version_dataset.py to watch snapshots + diffs in action.
• Run python examples/reproduce_experiment.py to see a generated dvc.yaml,

a lineage graph, and an environment-drift report.

Questions: support@datanest.dev.

Reproducibility Guide

"Reproducible" means another person (or future-you) can take your commit and your

recorded inputs and obtain the same result. In ML that requires four things

to be pinned together: the data, the code, the configuration, and the

environment. DVC pins the first three; this guide — and the

reproducibility.py module in this product — pins the fourth and stitches them

into a single run record.

1. The four axes of reproducibility

If any one of these is unpinned, "it worked last week" eventually becomes "I

can't reproduce my own number." The goal is to make every axis explicit and

recorded.

2. Seeds and determinism

Seed everything, once

Non-determinism most often enters through unseeded RNGs. set_global_seeds

seeds Python's random, plus NumPy and PyTorch if they are installed, and pins

PYTHONHASHSEED for child processes:

from data_versioning import set_global_seeds

seeded = set_global_seeds(42)
print(seeded)   # e.g. ['python.random', 'numpy', 'torch']

The PYTHONHASHSEED subtlety

Setting os.environ["PYTHONHASHSEED"] from inside a running interpreter does

not retroactively change this process's hash randomization — it only affects

interpreters started *afterward*. For fully deterministic set/dict iteration,

export it in your launcher before Python starts:

PYTHONHASHSEED=42 python src/train.py

set_global_seeds sets the variable so any subprocess you spawn inherits it, and

records the intent; the launcher line above closes the loop for the parent

process.

Sources of non-determinism that a seed does *not* fix

• GPU kernels. Many cuDNN/cuBLAS ops are nondeterministic by default. For

PyTorch, add torch.use_deterministic_algorithms(True) and set

CUBLAS_WORKSPACE_CONFIG=:4096:8. Expect a speed cost.

• Parallel data loading. Worker processes that shuffle without a per-worker

seed reorder batches. Seed each worker.

• Filesystem ordering. os.listdir() order is not guaranteed. Always

sorted() file lists before you build a dataset (the hashing.py walker does

this for you, which is why its tree_hash is stable across machines).

• Floating-point reduction order. Summing in a different order yields

bit-different results; multi-threaded BLAS can do this. Pin thread counts

(OMP_NUM_THREADS=1) when you need bit-exactness.

• Wall-clock timestamps and UUIDs written into outputs make every run look

"changed." Keep them out of artifacts you hash.

3. Capture the run environment

A seed makes a run repeatable *on the same machine*. To repeat it elsewhere you

also need the package versions, the interpreter, and the code commit. Capture

them into a record you commit next to your metrics:

from data_versioning import RunEnvironment

env = RunEnvironment.capture(
    seed=42,
    params={"n_estimators": 200, "max_depth": 8, "learning_rate": 0.05},
    track_packages=["numpy", "scikit-learn", "pandas"],  # only what matters
)
env.save("artifacts/run_env.json")

capture() records:

• Python version + implementation (CPython, PyPy, ...) and platform/arch.
• Installed package versions via importlib.metadata (optionally filtered to the

libraries you actually depend on — smaller, more meaningful records).

• The git commit and whether the working tree was dirty.
• Your seed and hyperparameters.

Anonymization note. Hostname capture is off by default because machine

names can leak organization details into shared artifacts. Pass

include_hostname=True only for private records.

4. Diff two environments to explain drift

The single most useful operation is diffing a fresh capture against the recorded

one. It turns "the number changed and I don't know why" into a precise list:

from data_versioning import RunEnvironment

recorded = RunEnvironment.load("artifacts/run_env.json")
current = RunEnvironment.capture(seed=42, params=recorded.params,
                                 track_packages=recorded.packages.keys())

diff = recorded.diff(current)
if not diff.is_reproducible:
    print(diff.summary())

Example output:

environment drift detected:
  python: 3.11.6 -> 3.12.1
  pkg ~  scikit-learn: 1.4.0 -> 1.5.1
  param  learning_rate: 0.05 -> 0.1

Now you know the model moved because of a Python upgrade, a scikit-learn bump,

and a changed learning rate — not gremlins.

Restoring what can be restored

restore() re-applies the seed immediately and returns warnings for the things

that cannot be fixed from inside a running process (you cannot downgrade a library

mid-run):

from data_versioning import restore
for warning in restore(recorded):
    print("WARN:", warning)

5. Bundle everything into a run manifest

A RunManifest is one JSON file that answers "what data, what code, what config,

what environment produced these metrics?" — the artifact you attach to a model in

your registry:

from data_versioning import RunManifest, hash_directory

manifest = RunManifest(run_id="run-2026-001", environment=env)
manifest.add_dataset("data/raw", hash_directory("data/raw").tree_hash)
manifest.add_dataset("data/features", hash_directory("data/features").tree_hash)
manifest.metrics = {"roc_auc": 0.913, "accuracy": 0.881}
manifest.save("artifacts/run_manifest.json")

Commit run_manifest.json to git. With the dataset hashes inside it you can

later assert that the data you are about to retrain on is *exactly* the data the

recorded result used.

6. Lineage: provenance and impact

When a number looks wrong, you need to know what produced it; when data changes,

you need to know what to rebuild. The lineage.py graph answers both:

from data_versioning import LineageGraph

g = LineageGraph()
g.add_stage("prepare", deps=["data/raw"], outs=["data/prepared"])
g.add_stage("train", deps=["data/prepared"], outs=["models/model.pkl"],
            metrics=["metrics.json"])

# "where did the model come from?" (upstream, in execution order)
print(g.provenance("dataset:models/model.pkl"))

# "if raw data changes, what must I recompute?" (everything downstream)
print(g.impacted_by("dataset:data/raw"))

# render it
open("lineage.dot", "w").write(g.to_dot())   # dot -Tpng lineage.dot -o lineage.png

Provenance is your audit trail; impact analysis is how you scope a re-run without

blindly rebuilding the world.

7. The reproducibility checklist

Before you publish a result, confirm every box:

• [ ] Data is pinned. dvc.lock is committed, or you have recorded the

tree_hash of every input dataset in the run manifest.

• [ ] Code is pinned. The git commit is clean (git_dirty is false in the

captured environment) and pushed.

• [ ] Config is pinned. Every hyperparameter lives in params.yaml (tracked

by DVC), not hard-coded in a script.

• [ ] Seeds are set at the very start of the run via set_global_seeds, and

PYTHONHASHSEED is exported in the launcher.

• [ ] Environment is captured with RunEnvironment.capture() and saved as an

artifact.

• [ ] Determinism is verified. Hash a key output directory twice with

hash_directory; the tree_hash must match. If it does not, fix the

non-determinism before trusting the number.

• [ ] The manifest is committed. One run_manifest.json links data + code +

config + environment + metrics.

8. A minimal reproducible run, start to finish

from data_versioning import (
    set_global_seeds, RunEnvironment, RunManifest, hash_directory,
)

SEED = 42
set_global_seeds(SEED)                                  # 1. determinism

params = {"n_estimators": 200, "max_depth": 8}          # 2. config
# ... run prepare -> featurize -> train -> evaluate ...
metrics = {"roc_auc": 0.913}

env = RunEnvironment.capture(seed=SEED, params=params,  # 3. environment
                             track_packages=["numpy", "scikit-learn"])

manifest = RunManifest(run_id="run-2026-001", environment=env)
manifest.add_dataset("data/raw", hash_directory("data/raw").tree_hash)  # 4. data
manifest.metrics = metrics
manifest.save("artifacts/run_manifest.json")            # 5. record it

Commit the manifest, git push, dvc push, and your result is reproducible by

anyone with read access to the repo and the remote.

Questions or feedback: support@datanest.dev.