← Back to all products

AI API Gateway

$39

Unified API for multiple LLM providers, rate limiting, fallback routing, response caching, and usage analytics dashboard.

📁 34 files
MarkdownYAMLPythonAzureLLMOpenAI

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 34 files

ai-api-gateway/ ├── LICENSE ├── README.md ├── configs/ │ └── gateway_config.yaml ├── free-sample.zip ├── guide/ │ └── 01-deployment-guide.md ├── guides/ │ └── deployment-guide.md ├── index.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── analytics.cpython-312.pyc │ │ ├── cache.cpython-312.pyc │ │ ├── client.cpython-312.pyc │ │ ├── rate_limiter.cpython-312.pyc │ │ ├── request_logger.cpython-312.pyc │ │ └── retry.cpython-312.pyc │ ├── analytics.py │ ├── cache.py │ ├── client.py │ ├── providers/ │ │ ├── __init__.py │ │ ├── __pycache__/ │ │ │ ├── __init__.cpython-312.pyc │ │ │ ├── anthropic_provider.cpython-312.pyc │ │ │ ├── local_provider.cpython-312.pyc │ │ │ └── openai_provider.cpython-312.pyc │ │ ├── anthropic_provider.py │ │ ├── local_provider.py │ │ └── openai_provider.py │ ├── rate_limiter.py │ ├── request_logger.py │ └── retry.py └── tests/ ├── __pycache__/ │ ├── conftest.cpython-312.pyc │ └── test_gateway.cpython-312.pyc ├── conftest.py └── test_gateway.py

📖 Documentation Preview README excerpt

AI API Gateway

Unified client for multiple LLM providers with rate limiting, fallback routing, response caching, request logging, and usage analytics.

Stop scattering OpenAI, Anthropic, and local model calls across your codebase. This gateway gives you a single complete() call that handles provider failover with circuit breakers, token-bucket rate limiting, SHA-256 response caching, structured JSONL logging, and usage analytics — with three provider adapters (OpenAI, Anthropic, OpenAI-compatible local servers) and a base class for adding more.

What You Get

  • Unified client — one complete() call for all providers, one chat() shortcut for single-turn conversations
  • Three provider adapters — OpenAI (SDK), Anthropic (SDK), and local/OpenAI-compatible servers (stdlib urllib, no dependencies)
  • Retry router with exponential backoff, jitter, and automatic fallback to the next provider in the chain
  • Circuit breaker that temporarily disables consistently-failing providers and re-tests them after a cooldown
  • Token bucket rate limiter (stdlib, thread-safe) with per-provider capacity and refill rates
  • Response cache with SHA-256 keys, TTL expiration, LRU eviction, and JSON persistence
  • Structured request logger writing JSONL with prompt previews, token counts, latency, and error details
  • Usage analytics with per-provider stats, per-model breakdowns, P95 latency, error rates, and Markdown report generation
  • YAML configuration for all gateway settings
  • Comprehensive test suite with mock/failing/flaky providers testing every component

File Tree


ai-api-gateway/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   ├── __init__.py                          # Package init
│   ├── client.py                            # GatewayClient — unified entry point
│   ├── rate_limiter.py                      # Token bucket rate limiter (stdlib)
│   ├── retry.py                             # RetryRouter + circuit breaker
│   ├── cache.py                             # GatewayCache — SHA-256 + TTL + LRU
│   ├── request_logger.py                    # Structured JSONL request logging
│   ├── analytics.py                         # Usage analytics + Markdown reports
│   └── providers/
│       ├── __init__.py                      # BaseProvider, CompletionRequest/Response, ProviderError
│       ├── openai_provider.py               # OpenAI Chat Completions adapter
│       ├── anthropic_provider.py            # Anthropic Messages adapter
│       └── local_provider.py                # OpenAI-compatible local server (stdlib urllib)
├── configs/
│   └── gateway_config.yaml                  # Full gateway configuration
├── tests/
│   ├── conftest.py                          # Path setup
│   └── test_gateway.py                      # Tests with mock/failing/flaky providers
└── guides/
    └── deployment-guide.md                  # Local, production, monitoring, troubleshooting

Quick Start

1. Install Dependencies


pip install -r requirements.txt

Only needed if you're using OpenAI or Anthropic providers. The local provider uses stdlib only.

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

src/analytics.py """ Usage Analytics — AI API Gateway ==================================== Aggregates request logs into actionable analytics: cost tracking, provider performance comparison, error rate monitoring, and usage trend detection. All analytics are computed from in-memory log entries and the JSONL log file — no external database required. By Datanest Digital — support@datanest.dev """ from __future__ import annotations import json import logging from collections import defaultdict from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path from typing import Any, Dict, List, Optional from src.request_logger import RequestLogEntry logger = logging.getLogger(__name__) @dataclass class ProviderStats: """Aggregate statistics for a single provider. Attributes: provider: Provider name. total_requests: Total number of requests. total_errors: Number of failed requests. total_cached: Number of cache hits. total_prompt_tokens: Sum of prompt tokens. total_completion_tokens: Sum of completion tokens. avg_latency_ms: Average request latency. p95_latency_ms: 95th percentile latency. error_rate: Fraction of requests that failed. """ provider: str = "" total_requests: int = 0 total_errors: int = 0 total_cached: int = 0 total_prompt_tokens: int = 0 total_completion_tokens: int = 0 avg_latency_ms: float = 0.0 p95_latency_ms: float = 0.0 # ... 178 more lines ...
Buy Now — $39 Back to Products