Getting Started with LLM Cost Optimizer

Track, analyse, and reduce LLM API spending with token accounting, intelligent model routing, response caching, budget alerts, and cost reporting.

Most teams discover they're spending 3-5x more than necessary on LLM APIs. This toolkit gives you per-request cost tracking, a model router that picks the cheapest model for each quality tier, exact-match response caching (typically 20-40% savings), budget alerting, and a report generator that produces HTML dashboards and Markdown summaries.

What You Get

• Token usage tracker with per-request logging, model breakdowns, and JSONL persistence
• Pricing table for OpenAI and Anthropic models with YAML-driven updates
• Model router that classifies prompts by complexity and selects the cheapest qualifying model
• Response cache with SHA-256 exact-match, TTL expiration, LRU eviction, and JSON persistence
• Budget monitor with daily/weekly/monthly thresholds, percentage-based warnings, and custom alert callbacks
• Token estimator with content-type detection for English prose, code, technical, and multilingual text
• Report generator producing both Markdown tables and a styled HTML cost dashboard

File Tree

llm-cost-optimizer/
├── README.md
├── LICENSE
├── requirements.txt
├── src/
│   ├── __init__.py
│   ├── token_tracker.py
│   ├── pricing.py
│   ├── model_router.py
│   ├── cache.py
│   ├── budget_alert.py
│   ├── report_generator.py
│   └── tokenizer_estimate.py
├── configs/
│   └── pricing.yaml
└── tests/
    ├── conftest.py
    └── test_optimizer.py

Quick Start

Install Dependencies

pip install -r requirements.txt

Only PyYAML is required (for pricing config). The core modules work without it using built-in defaults.

Track API Calls

from src.pricing import PricingTable
from src.token_tracker import TokenTracker

pricing = PricingTable("configs/pricing.yaml")
tracker = TokenTracker(log_path="usage.jsonl", pricing=pricing)

# After each API call, record the usage
tracker.record(
    model="gpt-4o",
    prompt_tokens=1200,
    completion_tokens=350,
    latency_ms=820,
    metadata={"feature": "customer_support", "user_id": "u-42"},
)

# Get aggregate stats
summary = tracker.get_summary()
print(f"Total cost: ${summary['total_cost_usd']:.4f}")
print(f"Total calls: {summary['total_events']}")

Core Features

Route to the Cheapest Model

The model router classifies prompts by complexity and selects the cheapest qualifying model automatically.

from src.model_router import ModelRouter

router = ModelRouter(pricing)

# The router classifies prompt complexity automatically
decision = router.route("Extract the email address from this text")
print(f"Use: {decision.selected_model}")    # e.g. "gpt-3.5-turbo" (tier 3)
print(f"Estimated cost: ${decision.estimated_cost:.6f}")

# Complex prompts get routed to premium models
decision = router.route("Analyze the trade-offs between microservices and monoliths")
print(f"Use: {decision.selected_model}")    # e.g. "gpt-4o" (tier 1)

# Set a hard budget cap per request
decision = router.route("Summarize this document", max_cost=0.001)

Cache Responses

Reduce costs by 20-40% with exact-match response caching.

from src.cache import ResponseCache

cache = ResponseCache(
    max_entries=10_000,
    default_ttl=3600,
    persist_path="cache.json",
)

# Before calling the API, check cache
cached = cache.get(prompt="What is RAG?", model="gpt-4o")
if cached:
    print("Cache hit — saved an API call")
else:
    response = call_your_llm(prompt)
    cache.put("What is RAG?", "gpt-4o", response)

print(f"Hit rate: {cache.stats['hit_rate']:.1%}")

Quality Tiers

The model router classifies prompts into three quality tiers:

Override automatic classification when you know the complexity:

decision = router.route(prompt, required_tier=3)  # force economy tier