← Back to all products

ML Data Versioning

$29

DVC setup, data pipeline versioning, experiment reproducibility, and artifact management workflows.

📁 17 files🏷 v1.0.0
JSONMarkdownPythonYAML

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 17 files

ml-data-versioning/ ├── LICENSE ├── README.md ├── dvc/ │ ├── dvc.yaml │ └── params.yaml ├── examples/ │ ├── reproduce_experiment.py │ └── version_dataset.py ├── guides/ │ ├── dvc-setup.md │ └── reproducibility-guide.md ├── src/ │ └── data_versioning/ │ ├── __init__.py │ ├── hashing.py │ ├── lineage.py │ ├── pipeline.py │ ├── reproducibility.py │ └── snapshots.py └── tests/ ├── test_hashing.py └── test_lineage.py

📖 Documentation Preview README excerpt

ML Data Versioning

A standard-library Python toolkit for **reproducible dataset versioning, lineage,

and DVC pipelines**. Hash datasets deterministically, snapshot and diff them

without any infrastructure, generate a valid dvc.yaml, build a provenance graph,

and capture the run environment that makes an experiment reproducible.

Every module is pure Python (no third-party runtime dependencies), so it runs

anywhere Python 3.9+ does and is fully unit-tested.

Why this exists

A versioned dataset is only half of reproducibility. To recreate a result you

need to pin four things together — the data, the code, the configuration, and

the environment. DVC handles the first three superbly; this toolkit fills the

gaps:

  • Before/without DVC — content-addressable hashing and a lightweight

snapshot store give you an exact "did this dataset change?" check with zero

setup (great for CI scratch space, notebooks, and audits).

  • Alongside DVC — generate dvc.yaml from code, render the pipeline as a

lineage graph for provenance/impact analysis, and capture the run environment

(seeds, params, package versions, git commit) that DVC does not record itself.

Features

  • Deterministic content hashing (hashing.py) — Merkle-style tree_hash

for a whole directory; identical bytes → identical hash on any OS, independent

of walk order and file mtimes. Streams large files; configurable ignore rules.

  • Dataset snapshots + diffs (snapshots.py) — capture, store, list, and

diff dataset versions in a tiny JSON-backed .snapshots/ registry, referenced

by label or hash prefix (git-style short SHAs).

  • Lineage graph (lineage.py) — a typed dataset → stage → output DAG with

topological ordering, cycle detection, provenance() (where did this come

from?) and impacted_by() (what must I recompute?), serializable to JSON and

Graphviz DOT.

  • DVC pipeline builder (pipeline.py) — define stages in code and emit a

valid dvc.yaml / params.yaml without PyYAML, including the `path:

{cache: false}` long form for metrics and plots.

  • Reproducibility capture (reproducibility.py) — seed every RNG, snapshot

the run environment, diff two environments to explain drift, and bundle

everything into a single RunManifest.

Quick start

No installation needed — the package is importable straight from src/.


# Run the test suite (standard library only; no pip installs required)
python -m unittest discover -s tests -v

# See dataset hashing, snapshots, and diffs in action
python examples/version_dataset.py

# See a generated dvc.yaml, a lineage graph, and an environment-drift report
python examples/reproduce_experiment.py

Use it in your own code:

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/data_versioning/hashing.py """ Content-addressable hashing for datasets and ML artifacts. This module computes deterministic, reproducible content hashes for individual files and whole directory trees. It is the foundation for dataset versioning: two byte-identical datasets always produce the same ``tree_hash`` regardless of the machine, the filesystem walk order, or the files' modification times. The approach mirrors how content-addressable stores work in practice (Git objects, DVC's cache): the *content* determines the address, not the path or the timestamp. A directory is hashed as a small Merkle-style manifest — every file is hashed independently, the (relpath, hash, size) triples are sorted into a canonical order, and the manifest itself is hashed to produce one stable ``tree_hash`` for the whole dataset. Design goals ------------ * **Deterministic** — identical bytes produce an identical hash on any OS, every run. ``os.walk`` order, inode numbers, and mtimes never leak into the result. * **Streaming** — large files are read in fixed-size chunks, so a multi-gigabyte parquet shard never has to fit in memory. * **Self-contained** — standard library only (``hashlib``, ``os``, ``fnmatch``). Typical use:: from data_versioning.hashing import hash_directory, diff_directories before = hash_directory("data/raw") print(before.short_hash, before.file_count, before.total_size) # ... the dataset changes on disk ... after = hash_directory("data/raw") delta = diff_directories(before, after) if delta.changed: print(delta.summary()) """ from __future__ import annotations import fnmatch import hashlib import os from dataclasses import dataclass from pathlib import Path from typing import Iterator, Sequence # Hash algorithms we accept. md5 is DVC's default cache key (chosen for speed, # not cryptographic strength); sha1/sha256 are offered for stronger collision # resistance when you care more about integrity than throughput. SUPPORTED_ALGORITHMS: tuple[str, ...] = ("md5", "sha1", "sha256") # ... 353 more lines ...
Buy Now — $29 Back to Products