Evaluation Guide — Computer Vision Toolkit

Understanding mAP, IoU, precision-recall, and segmentation metrics — with practical guidance on when each metric matters and how to interpret the numbers.

Metrics at a Glance

IoU — Intersection over Union

IoU is the foundation of all detection and segmentation metrics. It measures how well a predicted region overlaps with the ground truth.

IoU = Area(Prediction ∩ Ground Truth) / Area(Prediction ∪ Ground Truth)

Bounding Box IoU

from cv_toolkit.evaluation import compute_iou

# Perfect overlap → IoU = 1.0
iou = compute_iou([0, 0, 100, 100], [0, 0, 100, 100])

# Partial overlap → 0 < IoU < 1
iou = compute_iou([0, 0, 100, 100], [50, 50, 150, 150])

# No overlap → IoU = 0.0
iou = compute_iou([0, 0, 50, 50], [100, 100, 200, 200])

IoU Thresholds and What They Mean

Practical impact: At IoU=0.5, a detection that covers 50% of the object counts as correct. At IoU=0.75, the box must be much tighter. COCO's mAP@[.5:.95] averages across thresholds, rewarding models that produce tight boxes.

Segmentation IoU (Per-Pixel)

For segmentation, IoU is computed per-class at the pixel level:

from cv_toolkit.evaluation import compute_mask_iou, compute_mean_iou

# Predicted and ground truth masks (each pixel = class ID)
pred = [[0, 0, 1, 1],
        [0, 0, 1, 1],
        [2, 2, 2, 2],
        [2, 2, 2, 2]]

gt   = [[0, 0, 1, 1],
        [0, 1, 1, 1],
        [2, 2, 2, 2],
        [2, 2, 0, 2]]

# Per-class IoU
class_ious = compute_mask_iou(pred, gt, num_classes=3)
# {0: 0.6, 1: 0.8, 2: 0.857}

# Mean IoU across all classes
miou = compute_mean_iou(pred, gt, num_classes=3)

Average Precision (AP)

AP summarizes the precision-recall tradeoff for a detector at a specific IoU threshold. It answers: *"Across all confidence thresholds, how well does the model balance finding objects (recall) and avoiding false alarms (precision)?"*

How AP Is Computed

1. Sort all detections by confidence (highest first)

2. Walk through the sorted list. For each detection:

• If it matches a ground truth box (IoU ≥ threshold) → True Positive (TP)
• If it matches a GT box already matched → False Positive (FP)
• If no matching GT box → False Positive (FP)

3. At each step, compute:

• Precision = TP / (TP + FP) — "of my detections, how many are real?"
• Recall = TP / Total GT — "of all real objects, how many did I find?"

4. Plot the precision-recall curve

5. Compute the area under this curve = AP

Interpolation Methods

All-point interpolation (COCO style): compute the area under the smoothed PR curve using every unique recall value. More precise, the current standard.

11-point interpolation (Pascal VOC 2007): sample precision at 11 recall levels (0.0, 0.1, ..., 1.0) and average. Simpler but less accurate.

from cv_toolkit.evaluation import compute_ap_per_class, DetectionAnnotation

# Define ground truth and predictions
gt = [
    DetectionAnnotation("img1", class_id=0, box=[10, 10, 50, 50]),
    DetectionAnnotation("img1", class_id=0, box=[60, 60, 100, 100]),
]
preds = [
    DetectionAnnotation("img1", class_id=0, box=[12, 8, 48, 52], confidence=0.95),
    DetectionAnnotation("img1", class_id=0, box=[62, 58, 98, 102], confidence=0.80),
    DetectionAnnotation("img1", class_id=0, box=[200, 200, 250, 250], confidence=0.30),
]

ap, recalls, precisions = compute_ap_per_class(
    preds, gt, class_id=0, iou_threshold=0.5, method="all_points"
)
print(f"AP@0.5 for class 0: {ap:.4f}")

mAP — Mean Average Precision

mAP averages AP across all object classes. It is the single number that summarizes a detector's overall performance.

from cv_toolkit.evaluation import compute_map, compute_coco_map

# mAP at a single IoU threshold
result = compute_map(predictions, ground_truths, iou_threshold=0.5)
print(f"mAP@0.5: {result['mAP']}")
print(f"Per-class AP: {result['per_class_ap']}")

# COCO-style mAP across 10 thresholds (0.50, 0.55, ..., 0.95)
coco = compute_coco_map(predictions, ground_truths)
print(f"mAP@[.5:.95]: {coco['mAP@[.5:.95]']}")
print(f"mAP@.5:       {coco['mAP@.5']}")
print(f"mAP@.75:      {coco['mAP@.75']}")

Interpreting mAP Numbers

Important: mAP varies dramatically across datasets. An mAP@0.5 of 50% on COCO (80 classes, small objects, occlusion) is much harder to achieve than 90% on a custom dataset with 5 large, well-separated classes.

Classification Metrics

Accuracy

Accuracy = Correct Predictions / Total Predictions

When accuracy lies: With imbalanced classes (99% negative, 1% positive), a model that always predicts "negative" gets 99% accuracy while being completely useless. Use F1 or precision/recall instead.

Precision, Recall, F1

Precision = TP / (TP + FP)   — "When I say positive, am I right?"
Recall    = TP / (TP + FN)   — "Did I find all the positives?"
F1        = 2 × P × R / (P + R) — harmonic mean of precision and recall

from cv_toolkit.evaluation import compute_classification_metrics

metrics = compute_classification_metrics(
    y_true=[0, 0, 1, 1, 2, 2, 2],
    y_pred=[0, 1, 1, 1, 2, 0, 2],
    num_classes=3,
)

print(f"Overall accuracy: {metrics['accuracy']:.4f}")
for cls_id, cls_metrics in metrics['per_class'].items():
    print(f"  Class {cls_id}: P={cls_metrics['precision']:.3f} "
          f"R={cls_metrics['recall']:.3f} F1={cls_metrics['f1']:.3f}")

Confusion Matrix

The confusion matrix shows exactly where the model confuses classes:

             Predicted
             Cat  Dog  Bird
Actual Cat  [ 45   3    2 ]
       Dog  [  5  38    7 ]
       Bird [  1   4   45 ]

Read row-by-row: "Of 50 actual cats, the model predicted 45 as cat, 3 as dog, and 2 as bird."

High off-diagonal values reveal which class pairs are confused — this tells you where to focus data collection or model improvement.

Decision Guide: Which Metric to Report

Classification

• Balanced dataset: accuracy is fine as the primary metric
• Imbalanced dataset: report macro-F1 (averaged across classes) and per-class recall
• Binary classification: report precision, recall, and F1 for the positive class, plus the ROC-AUC if you have probability outputs
• Multi-label: report per-label AP and mean AP

Detection

• Academic paper / benchmark: report mAP@[.5:.95] (COCO standard)
• Production deployment: report mAP@0.5 (practical), plus per-class AP for your critical classes
• Safety-critical (autonomous driving, medical): report recall at high precision — missing a real object is worse than a false alarm

Segmentation

• Semantic segmentation: report mIoU and per-class IoU
• Instance segmentation: report mask mAP (like detection mAP but using mask IoU instead of box IoU)
• Panoptic segmentation: report Panoptic Quality (PQ)

Common Evaluation Mistakes

Mistake 1: Evaluating on Training Data

Never report metrics on data the model was trained on. Always hold out a validation set (10-20% of data) and ideally a test set that you only evaluate once.

Mistake 2: Ignoring Class Imbalance

If 95% of your images are "background" and 5% are "defect", a model that always predicts "background" gets 95% accuracy. Report per-class metrics and look at the minority class recall.

Mistake 3: Tuning on the Test Set

If you use test set metrics to choose hyperparameters, you're effectively training on the test set. Use a validation set for hyperparameter tuning and reserve the test set for final evaluation.

Mistake 4: Not Reporting Confidence Intervals

A single accuracy number is meaningless without context. Report the standard deviation across multiple runs (different seeds) or compute confidence intervals via bootstrapping.

Mistake 5: Wrong IoU Threshold for Your Application

Using mAP@0.5 when your application needs precise localization (surgery, manufacturing inspection) gives a misleadingly optimistic picture. Use a stricter threshold that matches your deployment requirements.

Training Guide — Computer Vision Toolkit

How to train classification, detection, and segmentation models with this toolkit, from data preparation through deployment.

Training Flow Overview

Raw Images → Dataset Loader → Augmentation → Model → Loss → Optimizer
                                                         ↓
                                  Checkpoint ← Scheduler ← Gradient Update
                                      ↓
                              Evaluation → mAP/IoU/Accuracy → Best Model

The toolkit handles every step after "Raw Images". Your job is to organize your data into the expected format, choose a config preset, and start training.

Step 1: Prepare Your Data

Classification

Organize images into class folders:

data/
├── train/
│   ├── class_a/
│   │   ├── img001.jpg
│   │   └── img002.jpg
│   └── class_b/
│       ├── img003.jpg
│       └── img004.jpg
└── val/
    ├── class_a/
    │   └── img005.jpg
    └── class_b/
        └── img006.jpg

from cv_toolkit.datasets import ImageFolderDataset

train_dataset = ImageFolderDataset(
    root="data/train",
    image_size=(224, 224),
)
print(f"Classes: {train_dataset.classes}")
print(f"Samples: {len(train_dataset)}")

Detection

Use COCO-format JSON annotations with one annotation file per split:

data/
├── images/
│   ├── train/
│   └── val/
└── annotations/
    ├── train.json    # COCO format
    └── val.json

from cv_toolkit.datasets import DetectionDataset

train_dataset = DetectionDataset(
    images_dir="data/images/train",
    annotations_file="data/annotations/train.json",
    annotation_format="coco",
)

Segmentation

Segmentation masks are single-channel images where each pixel value is a class ID:

data/
├── images/
│   ├── train/
│   └── val/
└── masks/
    ├── train/     # Same filenames as images, PNG format
    └── val/

Step 2: Choose a Training Config

Pick a preset from configs/training_config.yaml that matches your scenario:

Start with quick_experiment to verify your data loading, model forward pass, and loss computation are all working. Then switch to a full preset.

Step 3: Build Your Model

Classification

from cv_toolkit.classification import ClassificationPipeline

pipeline = ClassificationPipeline(
    backbone="resnet50",
    num_classes=10,
    pretrained=True,    # Use ImageNet weights
    freeze_backbone=False,
)
model = pipeline.model

Detection

from cv_toolkit.detection import DetectionModel, AnchorConfig

anchor_cfg = AnchorConfig(
    sizes=[32, 64, 128, 256, 512],
    aspect_ratios=[0.5, 1.0, 2.0],
)
model = DetectionModel(
    backbone="resnet50",
    num_classes=20,
    anchor_config=anchor_cfg,
)

Segmentation

from cv_toolkit.segmentation import SegmentationModel

model = SegmentationModel(
    backbone="resnet50",
    num_classes=21,
    decoder_type="unet",
)

Step 4: Train

from cv_toolkit.training import Trainer, TrainingConfig

config = TrainingConfig(
    num_epochs=30,
    learning_rate=1e-3,
    batch_size=32,
    mixed_precision=True,
    scheduler="cosine",
    warmup_epochs=3,
    checkpoint_dir="./checkpoints",
)

trainer = Trainer(model=model, config=config)

# fit() handles the full train/eval loop, checkpointing, and scheduling
history = trainer.fit(train_loader, eval_loader=val_loader)

What Happens During Training

Each epoch:

1. Forward pass — input images go through the model, producing predictions

2. Loss computation — compare predictions to ground truth labels/boxes/masks

3. Backward pass — compute gradients via backpropagation

4. Gradient accumulation — if enabled, accumulate over N batches before updating

5. Gradient clipping — prevent exploding gradients (especially important for detection)

6. Optimizer step — update model weights

7. LR scheduler step — adjust learning rate according to schedule

8. Evaluation — run on validation set without gradients, track best model

9. Checkpointing — save model state periodically and on improvement

Step 5: Monitor Training

Reading the Logs

Epoch 1/30 — loss: 2.3042, acc: 0.1024, lr: 1.00e-03, time: 45.2s | val_loss: 2.1850, val_acc: 0.1580
Epoch 2/30 — loss: 1.8534, acc: 0.3241, lr: 9.95e-04, time: 44.8s | val_loss: 1.6201, val_acc: 0.4012

What to watch:

• Train loss decreasing — the model is learning. If it plateaus early, increase LR or use a different scheduler.
• Val loss vs train loss — if train loss drops but val loss doesn't, you're overfitting. Add regularization (weight decay, augmentation, dropout).
• Val loss increasing — stop training. Your best checkpoint was saved automatically.
• LR schedule — cosine should smoothly decrease. Step should have visible drops at step boundaries.

Export History

# Save metrics as JSON for plotting
trainer.export_history("training_history.json")

Step 6: Evaluate

See the Evaluation Guide for detailed metric explanations.

from cv_toolkit.evaluation import compute_classification_metrics, compute_map

# Classification
metrics = compute_classification_metrics(y_true, y_pred, num_classes=10)
print(f"Accuracy: {metrics['accuracy']:.4f}")

# Detection
result = compute_map(predictions, ground_truths, iou_threshold=0.5)
print(f"mAP@0.5: {result['mAP']:.4f}")

Common Training Problems

Loss is NaN

Cause: Learning rate too high, or numeric overflow in loss computation.

Fix:

1. Reduce learning rate by 10x

2. Add gradient clipping (max_grad_norm: 1.0)

3. Disable mixed precision temporarily to isolate the issue

4. Check your data — NaN or infinite values in inputs will propagate

Loss Plateaus Immediately

Cause: Learning rate too low, or the model architecture doesn't have capacity.

Fix:

1. Try learning rates: 1e-1, 1e-2, 1e-3, 1e-4 (one order of magnitude at a time)

2. Verify data labels are correct (a random subset should make sense visually)

3. Start with a simpler model to confirm the data pipeline works

Validation Accuracy Much Lower Than Training

Cause: Overfitting.

Fix:

1. Add data augmentation (use classification_heavy preset)

2. Increase weight decay (try 1e-3 instead of 1e-4)

3. Use dropout in the classification head

4. Reduce model size (smaller backbone)

5. Get more training data

GPU Out of Memory

Fix:

1. Reduce batch size (use gpu_constrained preset)

2. Enable gradient accumulation to compensate

3. Enable mixed precision (mixed_precision: true)

4. Reduce input image size

5. Use a smaller backbone

Training Is Too Slow

Fix:

1. Enable mixed precision (20-50% speedup on modern GPUs)

2. Increase num_workers for data loading (set to number of CPU cores)

3. Use a pre-trained backbone with frozen layers (less computation)

4. Pin memory in DataLoader (pin_memory=True)

5. Ensure data is on local SSD, not network storage

Transfer Learning Strategy

Transfer learning from a pre-trained backbone almost always outperforms training from scratch, unless you have millions of labeled images.

When to freeze the backbone

Freeze when:

• Your dataset is small (<5000 images)
• Your images are similar to ImageNet (natural photos, animals, vehicles)
• You want fast training (only the head learns, much fewer parameters)

Don't freeze when:

• Your domain is very different from ImageNet (medical, satellite, industrial)
• You have enough data (>50000 images)
• You need maximum accuracy and have the compute budget

Two-stage fine-tuning

The most reliable transfer learning strategy:

# Stage 1: Train only the head (5-10 epochs)
for param in model.backbone.parameters():
    param.requires_grad = False
# Train with higher LR (1e-3)

# Stage 2: Unfreeze and train everything (20+ epochs)
for param in model.backbone.parameters():
    param.requires_grad = True
# Train with lower LR (1e-4) — don't destroy pre-trained features

Reproducibility Checklist

To reproduce training results exactly:

1. Set random seed in config (seed: 42)

2. Use deterministic operations (torch.use_deterministic_algorithms(True))

3. Fix num_workers — different worker counts can change data ordering

4. Pin the PyTorch version — numeric behavior changes between versions

5. Log the full config — save TrainingConfig alongside checkpoints

6. Save the data split — record which images went into train/val/test