← Back to all products

Data Labeling Pipeline

$29

Annotation workflow tools, quality assurance scripts, active learning selectors, and labeling interface templates.

📁 30 files
MarkdownYAMLPython

📄 Product Preview

Try the interactive reader and demo tools below, or get the full product with all content unlocked.

📖 Interactive Reader (Free Preview) ⚙ Try Demo Tools 📦 Download Free Sample

📁 File Structure 30 files

data-labeling-pipeline/ ├── LICENSE ├── README.md ├── configs/ │ ├── labeling_config.yaml │ └── schema_config.yaml ├── free-sample.zip ├── guide/ │ ├── 01-annotation-workflow.md │ └── 02-quality-assurance.md ├── guides/ │ ├── annotation-workflow.md │ └── quality-assurance.md ├── index.html ├── requirements.txt ├── src/ │ ├── __init__.py │ ├── __pycache__/ │ │ ├── __init__.cpython-312.pyc │ │ ├── active_learning.cpython-312.pyc │ │ ├── agreement.cpython-312.pyc │ │ ├── quality_audit.cpython-312.pyc │ │ ├── schema.cpython-312.pyc │ │ └── workflow.cpython-312.pyc │ ├── active_learning.py │ ├── agreement.py │ ├── quality_audit.py │ ├── schema.py │ └── workflow.py ├── templates/ │ └── labeling_interface.html └── tests/ ├── __pycache__/ │ ├── test_active_learning.cpython-312.pyc │ ├── test_agreement.cpython-312.pyc │ └── test_quality_audit.cpython-312.pyc ├── test_active_learning.py ├── test_agreement.py └── test_quality_audit.py

📖 Documentation Preview README excerpt

Data Labeling Pipeline

A complete toolkit for managing ML data labeling workflows — from annotator assignment through quality auditing to model-ready export. Includes inter-annotator agreement metrics (Cohen's κ, Fleiss' κ, Krippendorff's α), active learning sample selectors, label quality auditing, and a workflow orchestration engine.

Why This Exists

Labeling is the bottleneck of supervised ML. Most teams either:

  • Over-label: waste budget on redundant annotations without measuring agreement
  • Under-validate: ship noisy labels that poison model performance
  • Re-do work: discover label quality issues too late and re-label from scratch

This toolkit gives you the measurement and workflow tools to label efficiently, catch errors early, and know when you're done.

Features

  • Inter-annotator agreement — Cohen's κ (2 raters), Fleiss' κ (N raters), Scott's π, Krippendorff's α (with missing data support). All implemented in stdlib math — no scipy dependency for the core metrics.
  • Active learning selectors — Uncertainty, margin, entropy, random baseline, and query-by-committee strategies. Feed in predict_proba output, get back the most informative sample indices.
  • Label quality auditing — Annotator scorecards (agreement, speed, bias), suspect label detection, distribution drift testing (chi-squared), and automated audit reports.
  • Schema management — Define labeling taxonomies with categories, hierarchy, colors, and keyboard shortcuts. Validate annotations against schemas. Version and migrate schemas over time.
  • Workflow engine — Task creation, batch assignment (round-robin and difficulty-stratified), label submission, consensus resolution (majority vote / unanimous), adjudication, and training data export.
  • Labeling UI template — Ready-to-deploy HTML/CSS/JS annotation interface with keyboard shortcuts, progress tracking, and guidelines sidebar.

File Structure


data-labeling-pipeline/
├── README.md                          # This file
├── LICENSE                            # MIT License
├── requirements.txt                   # Python dependencies
├── src/
│   ├── __init__.py                    # Package metadata
│   ├── agreement.py                   # Cohen's κ, Fleiss' κ, Scott's π, Krippendorff's α
│   ├── active_learning.py             # Sample selection strategies
│   ├── quality_audit.py               # Annotator scoring, suspect detection, drift
│   ├── schema.py                      # Label schema definition and validation
│   └── workflow.py                    # Annotation workflow orchestration
├── tests/
│   ├── test_agreement.py              # Agreement metric tests
│   ├── test_active_learning.py        # Selector tests
│   └── test_quality_audit.py          # Audit function tests
├── configs/
│   ├── labeling_config.yaml           # Master workflow configuration
│   └── schema_config.yaml             # Label taxonomy definition
├── templates/
│   └── labeling_interface.html        # Browser-based annotation UI
└── guides/
    ├── annotation-workflow.md          # End-to-end workflow guide
    └── quality-assurance.md            # QA strategy and metrics guide

Quick Start

1. Compute Inter-Annotator Agreement


from src.agreement import cohens_kappa, fleiss_kappa

# Two raters
rater_a = ["pos", "neg", "pos", "neg", "pos", "neg", "pos", "neg"]
rater_b = ["pos", "neg", "neg", "neg", "pos", "neg", "pos", "pos"]

*... continues with setup instructions, usage examples, and more.*

📄 Code Sample .py preview

src/active_learning.py """ Active learning sample selectors — choose the most informative unlabeled samples to send to annotators next, so you label fewer items while training a better model. All selectors take a probability matrix (N_samples × N_classes) and return indices of the top-k most informative samples. The matrix typically comes from `model.predict_proba(X_unlabeled)`. Strategies implemented: 1. Uncertainty sampling — lowest max-class probability 2. Margin sampling — smallest gap between top-two probabilities 3. Entropy sampling — highest Shannon entropy 4. Random baseline — uniform random (for comparison) 5. Committee disagreement — query-by-committee via vote entropy All math uses stdlib only (math.log, sorted, etc.). """ from __future__ import annotations import logging import math import random from dataclasses import dataclass, field from typing import Callable, Dict, List, Optional, Sequence, Tuple logger = logging.getLogger(__name__) # --------------------------------------------------------------------------- # Type alias — a probability matrix is List[List[float]] where # probs[i][j] = P(class j | sample i) # --------------------------------------------------------------------------- ProbMatrix = List[List[float]] @dataclass class SelectionResult: """Returned by every selector: the chosen indices plus the score that ranked them, so you can inspect *why* each sample was selected.""" indices: List[int] scores: List[float] strategy: str pool_size: int budget: int def top(self, n: int = 5) -> List[Tuple[int, float]]: """Return top-n (index, score) pairs for quick inspection.""" # ... 335 more lines ...
Buy Now — $29 Back to Products