Batching and Scaling

Dynamic micro-batching is the single highest-leverage optimization for online

model serving. This guide explains how the batcher works, how to tune it, and

how to scale the service horizontally once a single replica is saturated.

Why batching wins

Frameworks predict on a *matrix* far more efficiently than on one row at a time.

Two overheads get amortized across the batch:

• Python / dispatch overhead. Each predict() call crosses into numpy / C /

BLAS once. A batch of 32 rows pays that crossing once; 32 single-row calls pay

it 32 times.

• Accelerator overhead. On a GPU/TPU, every call is a kernel launch plus a

host↔device copy. Tiny batches leave the device idle waiting on Python. Larger

batches keep it busy and can be 10–50× more throughput-efficient.

The catch: a REST endpoint receives requests one at a time. The batcher's job is

to *reassemble* a matrix from many small requests without making any single

caller wait too long.

The flush policy

A batch is dispatched as soon as either trigger fires:

flush when:  len(batch) >= max_batch_size        # the "size" trigger
        or   elapsed     >= max_batch_latency_ms  # the "latency" trigger

• The size trigger bounds how large a batch can get (memory, and the point

of diminishing returns on your hardware).

• The latency trigger bounds how long the *first* instance in a batch waits.

It is the key to keeping tail latency predictable under light load: if only

one request arrives, it waits at most max_batch_latency_ms and then goes.

In batching.py the loop grabs the first queued item, then keeps pulling more

with a shrinking asyncio.wait_for timeout until the deadline passes or the

size cap is hit. The Flask ThreadedBatcher does the same with a

queue.Queue(timeout=...).

Choosing max_batch_size and max_latency_ms

These two knobs are a throughput-vs-latency dial. Tune them empirically:

1. Measure single-row latency t1 and the per-row marginal cost in a

batch. Benchmark predict at sizes 1, 2, 4, 8, 16, 32, 64 and plot

rows/second. Throughput climbs, then plateaus — set max_batch_size near the

start of the plateau (often 16–64 for CPU sklearn, 64–512 for GPU deep nets).

2. Pick a latency budget. If your SLA is "p99 < 50 ms" and the model itself

takes ~20 ms on a full batch, you can afford max_batch_latency_ms of

roughly 10–15 ms of queueing. Smaller windows = lower latency, fewer co-batched

requests; larger windows = bigger batches, higher throughput, higher tail.

3. Validate under realistic load. Use examples/client_request.py's

concurrency burst, then read inference_batch_size from /metrics. If the

histogram still piles up at 1 under load, your latency window is too short (or

the load is genuinely too low to batch — which is fine).

Rules of thumb:

A little queueing theory

Little's Law (L = λ · W) is enough to sanity-check capacity. With arrival rate

λ (req/s) and mean time-in-system W (s), the average number of in-flight

requests is L. If your effective service rate (batches/s × batch_size) drops

below λ, the queue grows without bound and latency runs away. Two consequences:

• Headroom matters. Target sustained utilization well under 100% (say

≤ 70%). A service running at 95% utilization has no slack to absorb bursts and

will see latency explode at the first traffic spike.

• Watch queue depth, not just CPU. GET /healthz reports the batcher's

queue_depth. A steadily rising queue depth is the earliest, clearest signal

that you are under-provisioned — earlier than CPU saturation.

Scaling out

Once one replica is saturated, scale horizontally:

• More replicas. Each replica loads its own model copy and runs its own

batcher. Put a load balancer in front; round-robin is fine because the

batching happens *inside* each replica. This scales throughput linearly until

a shared bottleneck (database, feature store) appears.

• Workers vs replicas. On CPU, uvicorn --workers N gives N processes per

container, each with a full model copy — watch memory (N × model size). On GPU,

prefer one process per GPU and scale with more replicas/pods; multiple

workers contending for one device usually *hurts* throughput.

• Autoscale on the right signal. CPU utilization is a lagging, noisy signal

for batched inference. Prefer a custom metric: scale on queue_depth,

requests-per-second, or p95 of http_request_duration_seconds. In Kubernetes

this is an HPA on a Prometheus-adapter custom metric scraped from /metrics.

Memory, warmup, and cold starts

• Warm up the model. The first prediction often triggers lazy allocation

(BLAS threadpools, CUDA context, JIT). Send a synthetic batch during startup

so the first *real* request does not eat that cost. (The lifespan is the place

to add a warmup call.)

• Right-size memory limits. Account for: model size × workers, plus the peak

batch's intermediate tensors, plus framework overhead. Set container limits

with headroom; an OOM-kill mid-batch fails every co-batched caller at once.

• Pin thread counts on CPU. Oversubscribed BLAS threads (OMP_NUM_THREADS,

OPENBLAS_NUM_THREADS) across many workers cause thrashing. A common recipe is

one BLAS thread per process and concurrency from replicas instead.

Observability checklist

Everything you need to run this in production is already exported on /metrics:

• http_requests_total — request volume and error rate (by status).
• http_request_duration_seconds — latency histogram; compute p50/p95/p99 with

histogram_quantile() in PromQL.

• inference_batch_size — confirms batching is actually happening; if it stays

at 1 under load, revisit the latency window.

• inference_errors_total — model/inference failures, distinct from HTTP 4xx.
• queue_depth (via /healthz) — the leading indicator for autoscaling.

Alert on rising error rate, p99 latency breaching your SLA, and sustained queue

depth growth — in that order of urgency.

Serving Architecture

How the templates are wired, and the decisions behind them. This is the document

to read before you adapt the code to your own model and infrastructure.

The request lifecycle

A POST /predict call travels through five stages:

1. ASGI transport. Uvicorn accepts the connection and hands the request to

the FastAPI app. With --workers N you get N independent OS processes, each

with its own copy of the model and its own batcher.

2. Metrics middleware. MetricsMiddleware (in middleware.py) starts a

timer and, on the way out, records http_request_duration_seconds and

http_requests_total keyed by method, *route template*, and status. Using

the route template (/predict) instead of the raw path keeps metric

cardinality bounded.

3. Validation. Pydantic parses the body into a PredictRequest. Ragged or

empty instance matrices are rejected here with a 422 before any model code

runs — fail fast, fail cheap.

4. Batching. The handler calls batcher.submit_many(instances). Each

instance becomes its own future on the batcher's queue, so instances from

*different* concurrent requests are merged into one predict call. The

handler simply awaits its results.

5. Response. Predictions are assembled into a PredictResponse with the

serving model's name/version and the measured server-side latency.

Model loading happens in the lifespan, not per request

The model is loaded exactly once, during FastAPI's lifespan startup:

@asynccontextmanager
async def lifespan(app: FastAPI):
    handle = load_model(settings.model_path, fmt=settings.model_format, ...)
    batcher = DynamicBatcher(handle.predict_batch, ...)
    await batcher.start()
    app.state.handle = handle
    app.state.batcher = batcher
    yield
    await batcher.stop()

Loading in the lifespan (rather than at import or per request) means:

• Cold start is paid once. A 200 MB joblib artifact is deserialized a single

time per worker, not on the first request of every connection.

• Readiness is honest. Until the lifespan completes, the app is not marked

ready; an orchestrator that probes /healthz will not route traffic to a

worker whose model is still loading.

• Graceful shutdown is clean. On SIGTERM the lifespan's finally stops the

batcher, draining the in-flight batch before the process exits.

The module-level app = create_app() object is cheap to construct — it does no

I/O — so uvicorn model_serving.app:app is safe even though the model has not

loaded yet.

Liveness vs readiness

GET /healthz is deliberately a readiness probe: it returns 200 only when

the model handle is present *and* the batcher loop is running, and 503

otherwise. Wire it up accordingly:

A common mistake is pointing the *liveness* probe at a check that depends on the

model: if the model reload hiccups, Kubernetes will kill an otherwise healthy

pod. Keep liveness dumb and readiness smart.

The model abstraction

Every supported format collapses to a single ModelHandle with one method that

matters: predict_batch(rows) -> list. loaders.py provides:

• PickleLoader — any object with a .predict() method (stdlib pickle).
• JoblibLoader — scikit-learn / numpy estimators (lazy import joblib).
• OnnxLoader — ONNX graphs via onnxruntime (lazy import onnxruntime).

Because the heavy imports are lazy, importing model_serving to use just

the batcher never forces joblib or onnxruntime to be installed. To serve a

framework that is not built in (TorchScript, a TensorFlow SavedModel, a remote

model server), implement ModelLoader.load() and call register_loader():

from model_serving.loaders import ModelLoader, ModelHandle, register_loader

class TorchScriptLoader(ModelLoader):
    fmt = "torchscript"
    def load(self, path, *, name, version, metadata=None):
        import torch
        module = torch.jit.load(path).eval()
        def predict(rows):
            with torch.no_grad():
                return module(torch.tensor(rows)).tolist()
        return ModelHandle(name, version, self.fmt, predict, str(path))

register_loader("torchscript", TorchScriptLoader())

For tests and for models that are already in memory, skip files entirely with

ModelHandle.from_callable(fn, name=..., version=...) and pass it to

create_app(settings, model_handle=handle).

FastAPI vs Flask

Two transports ship with identical request/response contracts:

• app.py (FastAPI / ASGI) is the recommended path. Async I/O lets one

worker hold thousands of in-flight connections while the asyncio

DynamicBatcher merges their instances. CPU-bound predict calls run in a

thread executor so they never block the event loop.

• flask_app.py (Flask / WSGI) exists for teams standardized on Gunicorn.

WSGI is synchronous, so it uses the thread-based ThreadedBatcher — same

size/latency flush policy, different concurrency primitive.

Both reuse the *same* Pydantic models and the *same* metrics helpers, so you can

migrate between them without changing clients or dashboards.

A/B testing and canary releases

Routing between model versions is intentionally not baked into a single

process — that belongs at the load-balancer layer where it can be changed

without a redeploy. The pattern the templates support:

1. Run two deployments of the same image with different MODEL_PATH /

MODEL_VERSION (see docker/docker-compose.yml: serving-stable and

serving-canary).

2. Split traffic at your proxy (nginx split_clients, Envoy weighted clusters,

a Kubernetes Service mesh): e.g. 90% stable, 10% canary.

3. Compare each version's inference_* and http_request_duration_seconds

metrics. Promote the canary by shifting weights — no code change required.

Clients that must hit a specific version can pin it with the request body's

model_version field; a replica serving a different version answers 409 so a

mis-routed request fails loudly instead of returning silently wrong predictions.

Where to extend

• Auth: add an API-key/JWT dependency to /predict; leave /healthz and

/metrics open for probes and scrapers.

• Feature transforms: bake preprocessing into the saved pipeline (the

example uses a StandardScaler + LogisticRegression Pipeline) so serving

and training share the exact same transform.

• Multi-model: use ModelCache to keep several versions warm in one process

and select per request — useful when traffic per version is too low to justify

a dedicated deployment each.