Contents

Chapter 1

Deployment Guide

Local Development

bash
pip install -r requirements.txt
uvicorn src.serving_app:create_app --factory --reload --port 8000

Docker

dockerfile
FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ src/
COPY configs/ configs/
EXPOSE 8000
CMD ["uvicorn", "src.serving_app:create_app", "--factory", "--host", "0.0.0.0", "--port", "8000"]

Build and run:

bash
docker build -t model-server .
docker run -p 8000:8000 -v $(pwd)/models:/app/models model-server

Kubernetes

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: model-server
  template:
    metadata:
      labels:
        app: model-server
    spec:
      containers:
        - name: server
          image: your-registry.example.com/model-server:latest
          ports:
            - containerPort: 8000
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /ready
              port: 8000
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              memory: "512Mi"
              cpu: "500m"
            limits:
              memory: "2Gi"
              cpu: "2"

API Usage

Single Prediction

bash
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"features": {"age": 35, "income": 55000, "region": "north"}}'

Batch Prediction

bash
curl -X POST http://localhost:8000/predict/batch \
  -H "Content-Type: application/json" \
  -d '{"instances": [{"age": 35, "income": 55000}, {"age": 42, "income": 72000}]}'

A/B Test Override

bash
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -H "X-Model-Version: canary_model" \
  -d '{"features": {"age": 35, "income": 55000}}'
Chapter 2

Scaling Guide

Horizontal Scaling

Increase replicas in Kubernetes:

bash
kubectl scale deployment model-server --replicas=5

Vertical Scaling

For GPU models, request GPU resources:

yaml
resources:
  limits:
    nvidia.com/gpu: 1

Batching Strategies

StrategyLatencyThroughputWhen to Use
No batchingLowestLowestReal-time APIs, <10ms SLA
Client-side batchingMediumHighBulk scoring, nightly jobs
Server-side adaptiveMediumHighestGPU models, high traffic

For server-side adaptive batching, use BentoML or Triton (see configs/).

Monitoring

Key metrics to track:

  • p99 latency: Should stay under your SLA target
  • Error rate: Alert if >1% over 5 minutes
  • GPU utilization: If <50%, you're over-provisioned
  • Queue depth: If growing, add replicas

Caching

For models with repeated inputs (e.g. recommendation), add a cache:

python
from functools import lru_cache

@lru_cache(maxsize=10000)
def cached_predict(features_tuple):
    return model.predict(features_tuple)

Convert dicts to tuples for hashability.

Model Serving Toolkit v1.0.0 — Free Preview