LLM Evaluation Framework

$49

Automated evaluation harnesses, custom metrics, human feedback collection, regression testing, and quality monitoring dashboards.

📁 29 files

MarkdownPythonLLMOpenAI

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 29 files

llm-evaluation-framework/ ├── LICENSE ├── README.md ├── datasets/ │ ├── qa_sample.jsonl │ └── summarization_sample.jsonl ├── free-sample.zip ├── guide/ │ └── 01-evaluation-guide.md ├── guides/ │ └── evaluation-guide.md ├── index.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── dataset_runner.cpython-312.pyc │ │ ├── eval_harness.cpython-312.pyc │ │ ├── human_feedback.cpython-312.pyc │ │ ├── llm_judge.cpython-312.pyc │ │ ├── metrics.cpython-312.pyc │ │ ├── regression_tester.cpython-312.pyc │ │ └── report_generator.cpython-312.pyc │ ├── dataset_runner.py │ ├── eval_harness.py │ ├── human_feedback.py │ ├── llm_judge.py │ ├── metrics.py │ ├── regression_tester.py │ └── report_generator.py └── tests/ ├── __pycache__/ │ ├── test_harness.cpython-312.pyc │ └── test_metrics.cpython-312.pyc ├── test_harness.py └── test_metrics.py

📖 Documentation Preview README excerpt

LLM Evaluation Framework

A comprehensive toolkit for evaluating large language model outputs with automated

metrics, regression testing, LLM-as-judge scoring, and human feedback collection.

Whether you're comparing prompt templates, testing a model version upgrade, or

building a continuous evaluation pipeline, this framework gives you the metrics

library, harness infrastructure, and reporting tools to measure quality

systematically instead of eyeballing responses.

Features

13+ built-in metrics covering lexical (BLEU, ROUGE, F1), semantic (embedding

similarity), structural (JSON validity, format compliance), and LLM-as-judge

evaluation

Evaluation harness that loads JSONL/JSON/CSV datasets, runs model inference

with retry logic, computes metrics, and saves structured results

Regression tester that compares two evaluation runs, computes per-metric and

per-category deltas, and fires alerts when scores drop below configurable

thresholds

LLM-as-judge with pre-built rubrics (helpfulness, safety, code quality,

summarization) and pairwise comparison support

Human feedback collection with structured annotation schemas, JSONL storage,

and inter-annotator agreement metrics (Cohen's kappa)

Report generator producing self-contained HTML dashboards and Markdown

reports with metric visualizations, category breakdowns, and worst-sample

analysis

Dataset management with validation, stratified sampling, category splitting,

and data contamination detection

Quick Start

1. Run the metrics library standalone


from src.metrics import compute_all_metrics

prediction = "The capital of France is Paris."
reference = "Paris is the capital of France."

results = compute_all_metrics(prediction, reference)
for metric, score in results.items():
    print(f"  {metric}: {score:.4f}")

2. Run a full evaluation


from src.eval_harness import EvalHarness

def my_model(instruction: str, input_text: str = "") -> str:
    # Replace with your actual model call
    return "model response"

harness = EvalHarness(
    model_fn=my_model,
    metrics=["exact_match", "token_f1", "bleu", "rouge_l"],
    model_name="my-model-v1",
)


*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/dataset_runner.py """ Dataset Runner — LLM Evaluation Framework ============================================ Manages evaluation datasets: validation, splitting, sampling, and statistics. Separating dataset management from the eval harness keeps concerns clean. The harness calls model_fn and computes metrics; this module handles everything about the data itself — format validation, stratified sampling, train/test split verification, and dataset-level statistics. Supported dataset formats: - JSONL: One JSON object per line (primary format) - JSON: Array of objects - CSV: With headers matching expected field names """ from __future__ import annotations import csv import json import logging import math import random from collections import Counter from dataclasses import dataclass, field from io import StringIO from pathlib import Path from typing import Any, Dict, List, Optional, Sequence, Tuple logger = logging.getLogger(__name__) # ----------------------------------------------------------------------- # Dataset schema # ----------------------------------------------------------------------- # Required and optional fields for evaluation datasets. # The harness expects at least ``instruction``; everything else degrades # gracefully (e.g. no ``reference`` means reference-based metrics are skipped). REQUIRED_FIELDS = {"instruction"} OPTIONAL_FIELDS = {"reference", "expected", "input", "input_text", "category", "metadata"} ALL_KNOWN_FIELDS = REQUIRED_FIELDS | OPTIONAL_FIELDS @dataclass class DatasetStats: """Summary statistics for an evaluation dataset.""" total_samples: int = 0 categories: Dict[str, int] = field(default_factory=dict) has_reference: int = 0 # ... 364 more lines ...

Buy Now — $49 Back to Products