Features

This chapter covers the core features and capabilities of Model Evaluation Tool.

Features

• Core metrics — Accuracy, precision, recall, F1 score (macro and weighted)
• Confusion matrix — Multi-class confusion matrix with ASCII table rendering
• Per-class report — Precision, recall, F1, and support for every class
• Benchmark runner — Evaluate across multiple test splits with mean/std statistics
• File loaders — Load labels from JSONL, plain text, or structured JSON
• JSON export — Export full reports for dashboards and CI pipelines
• CLI interface — Evaluate, benchmark, and export from the terminal
• No dependencies — All math implemented from scratch using stdlib only

Quick Start

# Run demo with sample sentiment analysis data
python src/model_evaluation_tool.py --demo

# Evaluate predictions vs. labels
python src/model_evaluation_tool.py --predictions preds.txt --labels true.txt

# Export report as JSON
python src/model_evaluation_tool.py --predictions preds.txt --labels true.txt --export report.json

# Print confusion matrix
python src/model_evaluation_tool.py --confusion-matrix preds.txt true.txt

# Run benchmark across splits
python src/model_evaluation_tool.py --benchmark preds.txt true.txt --export bench.json

Project Structure

Follow this guide to get Model Evaluation Tool up and running in your environment.

Project Structure

model-evaluation-tool/
├── README.md
├── LICENSE
├── src/
│   └── model_evaluation_tool.py    # Core engine (~350 lines)
└── examples/
    ├── basic_usage.py               # Programmatic usage example
    └── sample_predictions.jsonl     # Sample prediction data

CLI Reference