← Back to all products

Model Evaluation Tool

$19

Python model evaluation suite with accuracy, precision, recall, F1, and per-class reports.

📁 11 files

MarkdownPython

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 11 files

model-evaluation-tool/ ├── LICENSE ├── README.md ├── examples/ │ ├── basic_usage.py │ └── sample_predictions.jsonl ├── free-sample.zip ├── guide/ │ ├── 01_features.md │ ├── 02_project-structure.md │ ├── 03_usage-examples.md │ └── 04_license.md ├── index.html └── src/ └── model_evaluation_tool.py

📖 Documentation Preview README excerpt

Model Evaluation Tool

Python model evaluation suite: accuracy, precision, recall, F1, confusion matrix, per-class reports, and benchmark runner. All metrics computed from scratch. Zero dependencies.

Part of the AI Toolkit collection by [CodeVault](https://ai-toolkit.codevault.dev).

Features

Core metrics — Accuracy, precision, recall, F1 score (macro and weighted)
Confusion matrix — Multi-class confusion matrix with ASCII table rendering
Per-class report — Precision, recall, F1, and support for every class
Benchmark runner — Evaluate across multiple test splits with mean/std statistics
File loaders — Load labels from JSONL, plain text, or structured JSON
JSON export — Export full reports for dashboards and CI pipelines
CLI interface — Evaluate, benchmark, and export from the terminal
No dependencies — All math implemented from scratch using stdlib only

Quick Start


# Run demo with sample sentiment analysis data
python src/model_evaluation_tool.py --demo

# Evaluate predictions vs. labels
python src/model_evaluation_tool.py --predictions preds.txt --labels true.txt

# Export report as JSON
python src/model_evaluation_tool.py --predictions preds.txt --labels true.txt --export report.json

# Print confusion matrix
python src/model_evaluation_tool.py --confusion-matrix preds.txt true.txt

# Run benchmark across splits
python src/model_evaluation_tool.py --benchmark preds.txt true.txt --export bench.json

Project Structure


model-evaluation-tool/
├── README.md
├── LICENSE
├── src/
│   └── model_evaluation_tool.py    # Core engine (~350 lines)
└── examples/
    ├── basic_usage.py               # Programmatic usage example
    └── sample_predictions.jsonl     # Sample prediction data

CLI Reference

Flag	Description
`--demo`	Run demo with built-in sample data
`--predictions FILE`	Predictions file (JSONL or plain text)
`--labels FILE`	True labels file (JSONL or plain text)
`--confusion-matrix PREDS LABELS`	Print confusion matrix
`--benchmark PREDS LABELS`	Run benchmark evaluation
`--export FILE`	Export report to JSON

... continues with setup instructions, usage examples, and more.

📄 Code Sample .py preview

examples/basic_usage.py#!/usr/bin/env python3 """ Basic usage example for the Model Evaluation Tool. Demonstrates: - Running a full evaluation with sample data - Inspecting per-class metrics - Building and printing a confusion matrix - Running benchmarks across multiple splits - Loading predictions from files """ import json import sys import tempfile from pathlib import Path # Allow running from the examples/ directory sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) from model_evaluation_tool import ( ConfusionMatrix, accuracy, benchmark, evaluate, f1_score, macro_f1, precision, recall, weighted_f1, ) def demo_basic_evaluation() -> None: """Run a full evaluation on a sentiment classification task.""" print("=== Basic Evaluation ===\n") # Simulated model predictions vs ground truth labels y_true = [ "positive", "negative", "positive", "neutral", "negative", "positive", "positive", "negative", "neutral", "positive", "negative", "neutral", "positive", "negative", "positive",

Buy Now — $19 Back to Products