← Back to all products

Model Serving Templates

$49

FastAPI/Flask model serving endpoints, batched inference, A/B testing, and canary deployment configurations.

📁 18 files🏷 v1.0.0
DockerJSONMarkdownYAMLPythonFastAPIFlaskPrometheus

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 18 files

model-serving-templates/ ├── LICENSE ├── README.md ├── docker/ │ ├── Dockerfile │ └── docker-compose.yml ├── examples/ │ ├── client_request.py │ └── serve_sklearn.py ├── guides/ │ ├── batching-and-scaling.md │ └── serving-architecture.md ├── src/ │ └── model_serving/ │ ├── __init__.py │ ├── app.py │ ├── batching.py │ ├── flask_app.py │ ├── loaders.py │ ├── middleware.py │ └── schemas.py └── tests/ ├── test_app.py └── test_batching.py

📖 Documentation Preview README excerpt

Model Serving Templates

Production-ready templates for serving machine-learning models over HTTP with

dynamic request micro-batching. Ships both a FastAPI (ASGI) and a Flask

(WSGI) implementation behind one identical API, plus a pluggable model loader

(pickle / joblib / ONNX), Prometheus metrics, runnable examples, tests, and a

slim Docker setup with a stable + canary topology.

The headline feature is the batcher: instances arriving in many concurrent

requests are transparently merged into single predict() calls, then fanned

back out to each caller — often a 10×+ throughput win with a bounded latency

cost you control.

Features

  • FastAPI app (src/model_serving/app.py) with POST /predict,

GET /healthz, GET /metrics, and lifespan-based model loading.

  • Dynamic micro-batcher (batching.py) — asyncio-based, with a size or

latency flush policy; pure stdlib and fully unit-tested.

  • Pluggable loaders (loaders.py) — pickle, joblib, and ONNX, with lazy

imports and a register_loader() hook for custom formats, plus an LRU

ModelCache for multi-version serving.

  • Pydantic v2 schemas (schemas.py) — request/response validation and

env-driven ServingSettings.

  • Metrics + logging (middleware.py) — a dependency-free Prometheus

exposition (counter + histogram) and an ASGI latency middleware.

  • Flask alternative (flask_app.py) — same API for Gunicorn/WSGI shops,

using a thread-based batcher.

  • Runnable examples, tests, and Docker — train-and-serve a real sklearn

model, an httpx client, pytest suites, a slim Dockerfile, and a compose file.

Project structure


model-serving-templates/
  src/model_serving/
    __init__.py        - package exports (create_app, DynamicBatcher, ...)
    app.py             - FastAPI app factory + /predict, /healthz, /metrics
    batching.py        - asyncio DynamicBatcher (size/latency flush)
    loaders.py         - ModelHandle + pickle/joblib/onnx loaders + ModelCache
    schemas.py         - Pydantic request/response models + ServingSettings
    middleware.py      - Prometheus metrics registry + ASGI MetricsMiddleware
    flask_app.py       - Flask/WSGI app + ThreadedBatcher
  examples/
    serve_sklearn.py   - train a sklearn model and boot the FastAPI app
    client_request.py  - httpx client + concurrency burst to exercise batching
  guides/
    serving-architecture.md   - request lifecycle, loading, A/B & canary
    batching-and-scaling.md    - tuning the batcher, autoscaling, observability
  tests/
    test_app.py        - FastAPI TestClient end-to-end tests
    test_batching.py   - stdlib-only batcher tests (also runnable standalone)
  docker/
    Dockerfile         - slim python:3.11 + uvicorn, non-root, healthcheck
    docker-compose.yml - stable + canary serving services
  README.md  LICENSE  manifest.json

Quick start

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/model_serving/app.py """FastAPI application factory for batched model serving. Endpoints --------- * ``POST /predict`` -- validate, micro-batch and run inference. * ``GET /healthz`` -- liveness/readiness probe (200 ready, 503 not ready). * ``GET /metrics`` -- Prometheus exposition of HTTP + inference metrics. The model is loaded once during the lifespan startup phase and shared across requests via ``app.state``; a :class:`~model_serving.batching.DynamicBatcher` runs in the background and merges concurrent ``/predict`` instances into single ``predict`` calls. Run it with:: MODEL_PATH=model.joblib uvicorn model_serving.app:app --host 0.0.0.0 --port 8000 """ from __future__ import annotations import logging import time from contextlib import asynccontextmanager from datetime import datetime, timezone from typing import Optional from fastapi import FastAPI, HTTPException, Request, Response, status from .batching import DynamicBatcher from .loaders import ModelHandle, load_model from .middleware import ( PROM_CONTENT_TYPE, MetricsMiddleware, MetricsRegistry, configure_logging, render_latest, ) from .schemas import HealthResponse, PredictRequest, PredictResponse, ServingSettings logger = logging.getLogger("model_serving.app") def _utcnow_iso() -> str: return datetime.now(timezone.utc).isoformat() def _build_handle(settings: ServingSettings) -> ModelHandle: """Load the configured model, raising a clear error if misconfigured.""" if not settings.model_path: raise RuntimeError( "MODEL_PATH is not set. Point it at a serialized model artifact, " "or pass model_handle=... to create_app() for in-memory models." # ... 138 more lines ...
Buy Now — $49 Back to Products