Deploying an LLM without systematic evaluation is like shipping software without
tests. It might work fine today, but you have no way to know when it breaks.
Models degrade silently — a prompt template change, a version bump, or new
training data can shift behavior in ways that aggregate accuracy numbers miss.
This guide walks through evaluation methodology: what to measure, how to
measure it, and how to set up continuous evaluation that catches regressions
before users do.
| Your Task | Primary Metrics | Secondary Metrics |
|---|---|---|
| Factual QA | exact_match, token_f1 | rouge_1, bleu |
| Open-ended generation | LLM-as-judge, human eval | rouge_l, format_compliance |
| Summarization | rouge_l, rouge_1, faithfulness | bleu, conciseness |
| Code generation | execution pass rate, exact_match | bleu, structural |
| Classification | exact_match, token_f1 | — |
| Translation | bleu, rouge_l | semantic_similarity |
Lexical metrics (BLEU, ROUGE, exact match) compare tokens. They're fast,
deterministic, and free. But they miss paraphrases: "Paris is the capital of
France" and "The French capital is Paris" score poorly against each other despite
being equivalent.
Semantic metrics (embedding similarity) capture meaning but require
embedding models, adding cost and a dependency. They're good for catching
paraphrases but can't distinguish factually correct from incorrect text
that happens to be semantically similar.
LLM-as-Judge captures nuance — coherence, helpfulness, safety — that
automated metrics miss. The tradeoff: cost (you pay for judge calls),
latency (seconds per evaluation), and reliability (judges have biases,
especially position bias and verbosity bias).
Recommendation: Use lexical metrics for CI/CD gate checks (fast, cheap),
LLM-judge for periodic deep evaluation, and human evaluation for launch
decisions.
differences.
per category.
A good evaluation dataset has:
1. Category coverage: Include all task types your model will handle.
If 30% of production traffic is code generation but your eval dataset is
100% factual QA, you won't catch code generation regressions.
2. Difficulty spread: Include easy, medium, and hard samples. A model that
scores 95% on easy samples but 20% on hard ones has a very different risk
profile than one scoring 70% across the board.
3. Edge cases: Deliberately include tricky inputs — ambiguous questions,
adversarial prompts, multi-step reasoning, long context.
4. Reference quality: Reference answers should be carefully vetted.
A bad reference makes every metric noisy.
Data contamination — where eval samples appear in training data — is the
most common source of inflated benchmark scores. Use the
train_test_overlap_check() function in dataset_runner.py to check for
exact-match overlap between your eval and training datasets.
Run on every model change (prompt update, version bump):
from src.eval_harness import EvalHarness
harness = EvalHarness(
model_fn=your_model,
metrics=["exact_match", "token_f1", "rouge_l"],
)
result = harness.run_dataset("datasets/qa_sample.jsonl")
# Fail the build if accuracy drops below threshold
assert result.aggregate_metrics["exact_match"] > 0.70, (
f"Accuracy too low: {result.aggregate_metrics['exact_match']}"
)Compare two model versions before promoting to production:
from src.regression_tester import RegressionTester
tester = RegressionTester()
result = tester.compare("results/v1.json", "results/v2.json")
if not result.passed:
print("Regressions detected:")
for alert in result.alerts:
if alert.severity == "error":
print(f" {alert.message}")Run weekly with LLM-as-judge for subjective quality:
from src.llm_judge import LLMJudge, HELPFULNESS_RUBRIC
judge = LLMJudge(
judge_fn=your_judge_model,
rubric=HELPFULNESS_RUBRIC,
)
result = judge.evaluate(
instruction="Explain quantum computing",
prediction=model_response,
reference=expert_answer,
)
print(f"Overall: {result.overall_score}")
for criterion, score in result.scores.items():
print(f" {criterion}: {score}")For launch decisions, collect human preferences:
from src.human_feedback import AnnotationSession, Annotation
session = AnnotationSession("model_comparison_v2_vs_v3")
# In your annotation UI:
session.add_annotation(Annotation.create_preference(
sample_id="sample_42",
annotator_id="reviewer_1",
preference="A", # Model A was better
duration_seconds=23.5,
))1. Baseline first: Run your current production model on the eval dataset.
This is your baseline.
2. Set warning at 5% below baseline: Small drops might be noise or
acceptable tradeoffs.
3. Set error at 10% below baseline: Drops this large almost always indicate
a real problem.
4. Per-category thresholds: Critical categories (e.g., safety-related
questions) should have tighter thresholds than nice-to-have categories.
With small eval datasets, score differences are often just noise. Rules of
thumb:
If you need rigorous significance testing, bootstrap the metric (resample
with replacement 1000 times, compute 95% confidence intervals).
Your eval dataset should include the hardest 20% of realistic queries. A model
that scores 95% on easy factual questions might score 30% on multi-hop reasoning.
BLEU was designed for machine translation and penalizes valid paraphrases.
For open-ended generation, ROUGE-L or LLM-as-judge are better choices.
A model that scores 5% higher but takes 3x longer might not be an improvement
in production. Always track latency alongside quality metrics.
If you tune prompts specifically to maximize eval scores, you're overfitting
to the eval set. Keep a held-out "canary" set that you never optimize against.
Evaluation should be continuous, not a one-time event. Set up automated eval
runs on every model change and review reports weekly.
┌─────────────────┐
│ Model Change │
│ (prompt/version)│
└────────┬────────┘
│
┌────────▼────────┐
│ CI Gate Check │ ← Fast lexical metrics
│ (auto, <1 min) │ Pass/fail threshold
└────────┬────────┘
│ pass
┌────────▼────────┐
│ Regression Test │ ← Compare to baseline
│ (auto, <5 min) │ Per-category analysis
└────────┬────────┘
│ pass
┌────────▼────────┐
│ LLM Judge Eval │ ← Weekly deep eval
│ (scheduled) │ Multi-criteria rubric
└────────┬────────┘
│
┌────────▼────────┐
│ Human Review │ ← Before major releases
│ (manual) │ Pairwise preferences
└────────┬────────┘
│
┌────────▼────────┐
│ Deploy │
└─────────────────┘evaluation of language models
benchmark with LLM judges
Open-source evaluation framework for few-shot tasks