Contents

Chapter 1

Deployment Guide — AI API Gateway

A practical guide to deploying the gateway in different environments, from local development to production.

Architecture Overview

  Your Application
        │
        ▼
┌───────────────────────────────────────────────┐
│              GatewayClient                    │
│                                               │
│  ┌──────────┐  ┌──────────┐  ┌────────────┐  │
│  │  Cache    │  │  Rate    │  │  Request   │  │
│  │  Layer    │  │  Limiter │  │  Logger    │  │
│  └──────────┘  └──────────┘  └────────────┘  │
│                                               │
│  ┌────────────────────────────────────────┐   │
│  │          Retry Router                  │   │
│  │  ┌──────┐  ┌──────────┐  ┌──────┐     │   │
│  │  │OpenAI│→ │Anthropic │→ │Local │     │   │
│  │  └──────┘  └──────────┘  └──────┘     │   │
│  └────────────────────────────────────────┘   │
│                                               │
│  ┌────────────────────────────────────────┐   │
│  │          Usage Analytics               │   │
│  └────────────────────────────────────────┘   │
└───────────────────────────────────────────────┘

Local Development

Minimal Setup (One Provider)

python
from src.client import GatewayClient
from src.providers.openai_provider import OpenAIProvider

gateway = GatewayClient(
    providers=[OpenAIProvider(api_key="YOUR_API_KEY_HERE")],
    enable_logging=False,  # Less noise during development
)

answer = gateway.chat("What is the meaning of life?")
print(answer)

With a Local LLM Server

If you're running a local model (Ollama, vLLM, llama.cpp), use the LocalProvider:

python
from src.providers.local_provider import LocalProvider

# Ollama
gateway = GatewayClient(
    providers=[LocalProvider(base_url="http://localhost:11434/v1", default_model="llama3")],
    enable_rate_limiting=False,  # Local server, no API quota
)

# vLLM
gateway = GatewayClient(
    providers=[LocalProvider(base_url="http://localhost:8000/v1")],
)

Production Deployment

Multi-Provider with Fallback

python
from src.client import GatewayClient
from src.providers.openai_provider import OpenAIProvider
from src.providers.anthropic_provider import AnthropicProvider
from src.providers.local_provider import LocalProvider

gateway = GatewayClient(
    providers=[
        # Primary — fastest and cheapest for most tasks
        OpenAIProvider(
            api_key=os.environ["OPENAI_API_KEY"],
            default_model="gpt-4o-mini",
        ),
        # Fallback 1 — used when OpenAI is down or rate-limited
        AnthropicProvider(
            api_key=os.environ["ANTHROPIC_API_KEY"],
            default_model="claude-sonnet-4-20250514",
        ),
        # Fallback 2 — local model for total API outage
        LocalProvider(
            base_url="http://internal-llm.example.com:8000/v1",
        ),
    ],
    enable_cache=True,
    cache_path="/var/lib/gateway/cache.json",
    log_path="/var/log/gateway/requests.jsonl",
    max_retries=3,
)

Environment Variables

Never hardcode API keys. Use environment variables:

bash
export OPENAI_API_KEY="sk-YOUR_KEY_HERE"
export ANTHROPIC_API_KEY="sk-ant-YOUR_KEY_HERE"

Rate Limiting Configuration

Set per-provider rate limits to stay within your API quota:

python
gateway = GatewayClient(providers=[])

# OpenAI Tier 2: ~500 RPM → ~8 RPS
gateway.add_provider(
    OpenAIProvider(api_key=os.environ["OPENAI_API_KEY"]),
    rate_limit_capacity=30,   # Burst of 30
    rate_limit_refill=8.0,    # 8 req/sec sustained
)

# Anthropic: ~1000 RPM → ~16 RPS
gateway.add_provider(
    AnthropicProvider(api_key=os.environ["ANTHROPIC_API_KEY"]),
    rate_limit_capacity=50,
    rate_limit_refill=16.0,
)

Monitoring

Log Analysis

The gateway writes structured JSONL logs. Analyse them with standard tools:

bash
# Count requests by provider
jq -r '.provider' gateway_requests.jsonl | sort | uniq -c | sort -rn

# Find slow requests (>2s)
jq 'select(.latency_ms > 2000)' gateway_requests.jsonl

# Count errors
jq 'select(.error != "")' gateway_requests.jsonl | wc -l

# Cost estimate (rough)
jq '[.prompt_tokens + .completion_tokens] | add' gateway_requests.jsonl

Analytics Reports

Generate analytics reports programmatically:

python
from src.analytics import UsageAnalytics

analytics = UsageAnalytics()
analytics.load_from_file("gateway_requests.jsonl")

# Per-provider breakdown
for stat in analytics.by_provider():
    print(f"{stat.provider}: {stat.total_requests} reqs, "
          f"{stat.error_rate:.1%} errors, "
          f"{stat.avg_latency_ms:.0f}ms avg")

# Full Markdown report
report = analytics.summary_markdown()
with open("analytics_report.md", "w") as f:
    f.write(report)

Health Checks

Use in a health check endpoint or monitoring script:

python
health = gateway.health()
# {'openai': True, 'anthropic': True, 'local': False}

provider_health = gateway.provider_health
# {'openai': {'total_requests': 1024, 'total_failures': 3, 'error_rate': 0.003, ...}}

Circuit Breaker Behaviour

The retry router includes a circuit breaker that protects against cascading failures:

1. Closed (normal): Requests flow through normally.

2. Open (tripped): After N consecutive failures (default: 5), the circuit opens. No requests are sent to the provider for a cooldown period (default: 60s).

3. Half-open (testing): After the cooldown, the next request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit re-opens.

Normal → 5 failures → Open (60s) → Half-open → Success → Normal
                                              → Failure → Open (60s)

Scaling Considerations

ConcernSolution
Cache sizeSet max_entries based on memory budget. 5K entries ≈ 50-100MB.
Log file sizeRotate gateway_requests.jsonl with logrotate or a cron job.
Thread safetyThe rate limiter is thread-safe. The cache and logger use append-only writes.
Multiple processesEach process gets its own cache and rate limiter state. For shared state, use Redis (not included).

Troubleshooting

"Rate limit timeout" errors

  • Increase rate_limit_capacity or rate_limit_refill for the provider.
  • Check your API tier and upgrade if needed.

All providers failing

  • Run gateway.health() to check provider connectivity.
  • Verify API keys are set correctly.
  • Check the JSONL log for error details.

Cache not helping

  • Check gateway.cache_stats for hit rate.
  • If hit rate is low, your queries may be too unique (each prompt is different).
  • Try setting cache_temperature_zero_only=True and using temperature=0 for deterministic queries.
AI API Gateway v1.0.0 — Free Preview