← Back to all products

Model Serving Toolkit

$39

Deploy models with FastAPI, TensorFlow Serving, Triton, and BentoML. Includes A/B testing and canary patterns.

📁 36 files
MarkdownYAMLPythonDockerKubernetesFastAPI

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 36 files

model-serving-toolkit/ ├── LICENSE ├── README.md ├── configs/ │ ├── bentoml_service.yaml │ ├── serving_config.yaml │ ├── tf_serving.yaml │ └── triton_config.yaml ├── free-sample.zip ├── guide/ │ ├── 01-deployment-guide.md │ └── 02-scaling-guide.md ├── guides/ │ ├── deployment-guide.md │ └── scaling-guide.md ├── index.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── health.cpython-312.pyc │ │ ├── inference.cpython-312.pyc │ │ ├── middleware.cpython-312.pyc │ │ ├── model_loader.cpython-312.pyc │ │ ├── routing.cpython-312.pyc │ │ ├── serving_app.cpython-312.pyc │ │ └── validation.cpython-312.pyc │ ├── health.py │ ├── inference.py │ ├── middleware.py │ ├── model_loader.py │ ├── routing.py │ ├── serving_app.py │ └── validation.py └── tests/ ├── __init__.py ├── __pycache__/ │ ├── __init__.cpython-312.pyc │ ├── test_routing.cpython-312.pyc │ └── test_serving.cpython-312.pyc ├── test_routing.py └── test_serving.py

📖 Documentation Preview README excerpt

Model Serving Toolkit

Deploy ML models with FastAPI, A/B testing, canary routing, and multi-backend support.

A complete model serving solution with request validation, health probes, traffic routing, and configuration templates for TensorFlow Serving, Triton, and BentoML.

What You Get

  • FastAPI serving application with single and batch prediction endpoints
  • Model loader abstraction supporting pickle, JSON, and ONNX formats with hot-swap capability
  • A/B testing router with weighted traffic splits, sticky sessions, and header-based overrides
  • Canary deployment support with dynamic weight updates for gradual rollouts
  • Request validation with schema-based type checking, range validation, and allowed-value constraints
  • Health and readiness probes following Kubernetes patterns with serving metrics
  • Request logging middleware with per-request latency tracking and distributed tracing
  • Inference engine with pre/post-processing hooks, batch support, and error isolation
  • Serving configs for TensorFlow Serving, NVIDIA Triton, and BentoML
  • Deployment guides for Docker, Kubernetes, and scaling strategies

Quick Start


pip install -r requirements.txt
uvicorn src.serving_app:create_app --factory --reload --port 8000

Then test with:


curl http://localhost:8000/health
curl http://localhost:8000/models

File Tree


model-serving-toolkit/
├── README.md
├── LICENSE
├── requirements.txt
├── .gitignore
├── src/
│   ├── __init__.py
│   ├── serving_app.py        # FastAPI application
│   ├── model_loader.py       # Model loading + hot-swap
│   ├── inference.py          # Single + batch inference engine
│   ├── routing.py            # A/B testing + canary routing
│   ├── validation.py         # Request schema validation
│   ├── health.py             # Liveness/readiness + metrics
│   └── middleware.py         # Request logging middleware
├── configs/
│   ├── serving_config.yaml   # Main serving configuration
│   ├── tf_serving.yaml       # TensorFlow Serving config
│   ├── triton_config.yaml    # NVIDIA Triton config
│   └── bentoml_service.yaml  # BentoML service config
├── tests/
│   ├── __init__.py
│   ├── test_serving.py       # Component tests
│   └── test_routing.py       # Routing tests

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/health.py """ Health and Readiness Endpoints ============================== Implements the Kubernetes-style liveness/readiness probe pattern. - /health (liveness): Returns 200 if the process is alive. - /ready (readiness): Returns 200 only when all models are loaded and the service is ready to accept inference requests. - /metrics: Returns basic serving metrics (request count, latency). """ from __future__ import annotations import logging import time from dataclasses import dataclass, field from typing import Any, Dict, List logger = logging.getLogger(__name__) @dataclass class ServingMetrics: """Tracks inference serving metrics in memory.""" total_requests: int = 0 total_errors: int = 0 total_latency_ms: float = 0.0 model_request_counts: Dict[str, int] = field(default_factory=dict) model_error_counts: Dict[str, int] = field(default_factory=dict) model_latency_totals: Dict[str, float] = field(default_factory=dict) _start_time: float = field(default_factory=time.time) def record_request(self, model_name: str, latency_ms: float, error: bool = False) -> None: """Record a single inference request.""" self.total_requests += 1 self.total_latency_ms += latency_ms self.model_request_counts[model_name] = self.model_request_counts.get(model_name, 0) + 1 self.model_latency_totals[model_name] = self.model_latency_totals.get(model_name, 0.0) + latency_ms if error: self.total_errors += 1 self.model_error_counts[model_name] = self.model_error_counts.get(model_name, 0) + 1 def to_dict(self) -> Dict[str, Any]: uptime = time.time() - self._start_time avg_latency = self.total_latency_ms / self.total_requests if self.total_requests > 0 else 0.0 return { "uptime_seconds": round(uptime, 1), "total_requests": self.total_requests, "total_errors": self.total_errors, # ... 72 more lines ...
Buy Now — $39 Back to Products