← Back to all products

ML Pipeline Templates

$49

End-to-end ML pipelines: data ingestion, preprocessing, training, evaluation, and deployment orchestration.

📁 18 files🏷 v1.0.0
JSONMarkdownPythonYAMLAirflow

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 18 files

ml-pipeline-templates/ ├── LICENSE ├── README.md ├── configs/ │ └── pipeline.yaml ├── examples/ │ ├── run_pipeline.py │ └── sklearn_pipeline.py ├── guides/ │ ├── orchestration-options.md │ └── pipeline-architecture.md ├── requirements.txt ├── src/ │ └── ml_pipeline/ │ ├── __init__.py │ ├── evaluation.py │ ├── ingestion.py │ ├── pipeline.py │ ├── preprocessing.py │ ├── steps.py │ └── training.py └── tests/ ├── test_pipeline.py └── test_preprocessing.py

📖 Documentation Preview README excerpt

ML Pipeline Templates

End-to-end, dependency-light templates for building tabular machine-learning

pipelines: data ingestion -> preprocessing -> training -> evaluation, chained

together by a small DAG orchestrator you can actually read.

The orchestration core is pure standard library, so it is fast to import and

trivial to test. The ML stages use pandas/numpy and prefer scikit-learn when

it is installed -- but every stage has a numpy fallback, so the whole pipeline

(and the test suite) runs end-to-end even with nothing but pandas + numpy.

Features

  • DAG orchestrator (Pipeline / Step) -- declare dependencies, get

deterministic topological execution, per-step timing/status, cycle and

missing-dependency detection, and fail_fast handling. No third-party deps.

  • Ingestion into one Dataset interface from CSV, Parquet, or SQL (or an

in-memory DataFrame), so downstream stages never touch a file path.

  • Preprocessing built on a scikit-learn ColumnTransformer (impute + scale +

one-hot encode), with a dependency-free PandasFeaturePipeline fallback and a

leakage-safe train/test split (optionally stratified).

  • Training with backend-agnostic K-fold cross-validation that works with any

fit/predict estimator -- scikit-learn models or the bundled numpy

baselines (RidgeRegressor, MeanRegressor, MajorityClassifier).

  • Evaluation with pure-numpy regression and classification metrics plus a

renderable EvaluationReport (Markdown, including a confusion matrix).

  • Config-driven -- build the standard four-stage pipeline from

configs/pipeline.yaml; retarget your data without editing Python.

  • Tested -- a pytest suite covers DAG ordering, context passing, error

handling, and the preprocessing transforms.

Requirements

  • Python 3.10+
  • pandas, numpy, PyYAML (required by the ML stages)
  • scikit-learn (recommended; auto-used when present), pyarrow (for Parquet) --

both optional


pip install -r requirements.txt

Quick start

Everything runs straight from the product folder -- the examples and tests add

src/ to the path, so no install step is needed.


# 1) Run the full pipeline on a generated toy dataset
python examples/run_pipeline.py

# 2) Run it from the YAML config instead of an inline config
python examples/run_pipeline.py --config configs/pipeline.yaml

# 3) See the scikit-learn ColumnTransformer + RandomForest path
python examples/sklearn_pipeline.py        # prints an install hint if sklearn is absent

# 4) Run the test suite
pip install pytest
python -m pytest tests/ -v

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/ml_pipeline/pipeline.py"""A minimal directed-acyclic-graph (DAG) orchestrator for ML pipelines. :class:`Pipeline` takes a set of :class:`ml_pipeline.steps.Step` objects, derives a dependency graph from each step's declared ``inputs``, computes a deterministic topological execution order (Kahn's algorithm), and runs the steps in that order while threading a shared ``context`` dict (step name -> output) through them. This module has **no third-party dependencies** -- it is pure standard library, so the orchestration core stays importable in any environment, including ones without pandas or scikit-learn. The optional :func:`build_standard_pipeline` helper wires the four bundled ML steps from a config dict; :func:`load_config` reads such a dict from a YAML file (PyYAML imported lazily, only when used). """ from __future__ import annotations import heapq import logging import time from dataclasses import dataclass, field from typing import Any, Iterable from .steps import ( EvaluationStep, IngestionStep, PreprocessingStep, Step, StepResult, StepStatus, TrainingStep, ) logger = logging.getLogger("ml_pipeline") class PipelineError(RuntimeError): """Base class for all pipeline construction / execution errors.""" class MissingDependencyError(PipelineError):
Buy Now — $49 Back to Products