Data Labeling Pipeline

Chapter 1

Annotation Workflow Guide

Overview

This guide walks you through setting up and running a complete data labeling workflow — from raw unlabeled data to model-ready training labels.

The Labeling Lifecycle

┌─────────────┐    ┌──────────────┐    ┌───────────────┐    ┌──────────┐
│ Raw Data    │ →  │ Task Queue   │ →  │ Annotation    │ →  │ Consensus│
│ (unlabeled) │    │ (prioritised)│    │ (N annotators)│    │ Resolution│
└─────────────┘    └──────────────┘    └───────────────┘    └──────────┘
                                                                  │
┌─────────────┐    ┌──────────────┐    ┌───────────────┐         │
│ Training    │ ←  │ Export       │ ←  │ Quality Audit │ ←───────┘
│ Dataset     │    │ (approved)   │    │ & Adjudication│
└─────────────┘    └──────────────┘    └───────────────┘

Step-by-Step Setup

1. Define Your Schema

Before anything else, decide what labels you need. Open configs/schema_config.yaml and define:

Categories: What are the possible labels?
Descriptions: How should annotators interpret each category?
Examples: What are clear-cut examples of each?
Hierarchy: Do labels have parent-child relationships?

python

from src.schema import LabelSchema, Category, LabelType

schema = LabelSchema(
    name="my_project",
    version="1.0",
    label_type=LabelType.CLASSIFICATION,
    categories=[
        Category(
            name="positive",
            description="Expresses satisfaction or approval",
            examples=["Great product!", "Highly recommend"],
            counter_examples=["It's okay (→ neutral)"],
        ),
        # ... more categories
    ],
)

Key decisions:

Start with fewer categories. You can always split later.
Write descriptions as if explaining to a new hire who's never seen your data.
Include counter-examples for every common mistake.

2. Configure the Workflow

Edit configs/labeling_config.yaml. The critical settings:

Setting	Recommendation	Why
`redundancy`	3 for classification, 5 for subjective	More raters = better agreement measurement
`assignment_strategy`	`stratified`	Routes hard items to experienced annotators
`consensus.min_agreement`	0.67 (2/3)	Balances quality vs. throughput
`quality.golden_items.fraction`	0.10	Catches random clicking without excessive cost
`active_learning.strategy`	`entropy`	Best general-purpose uncertainty measure

3. Prepare Your Data

Create Task objects from your unlabeled data:

python

from src.workflow import Task, WorkflowEngine

tasks = [
    Task(
        task_id=f"item_{i:04d}",
        data={"text": row["text"]},
        difficulty="normal",  # or "easy" / "hard"
    )
    for i, row in enumerate(your_data)
]

engine = WorkflowEngine(redundancy=3, min_agreement=0.67)
engine.add_tasks(tasks)

4. Create Batches and Assign

python

annotator_ids = ["ann_alice", "ann_bob", "ann_charlie"]

batch = engine.create_batch(
    batch_id="batch_001",
    annotator_ids=annotator_ids,
    strategy="round_robin",
    max_tasks=50,
)

5. Collect Labels

As annotators work (through your UI or the included templates/labeling_interface.html):

python

engine.submit_label(
    assignment_id="asgn_abc123",
    label="positive",
    duration_sec=4.2,
)

6. Resolve Consensus

Once all annotators have completed their assignments:

python

results = engine.resolve_all()

for result in results:
    if result.needs_adjudication:
        print(f"  {result.task_id} needs expert review")
    else:
        print(f"  {result.task_id} → '{result.final_label}'")

7. Adjudicate Disagreements

Tasks that can't reach consensus go to a senior annotator:

python

for task in engine.tasks_needing_adjudication:
    # Show the task + all submitted labels to the expert
    engine.adjudicate(task.task_id, "neutral", adjudicator_id="expert_01")

8. Run Quality Audit

python

from src.quality_audit import generate_audit_report, Annotation

# Convert engine's internal data to Annotation objects for auditing
annotations = [...]  # build from engine._assignments
report = generate_audit_report(annotations)
print(report)

9. Export Training Data

python

approved_data = engine.export_approved()
# → [{"task_id": ..., "data": {...}, "label": ..., "metadata": {...}}, ...]

Active Learning Loop

For maximum labeling efficiency, use the active learning module:

┌─────────────────────────────────────────────────┐
│                                                 │
│   ┌──────────┐    ┌───────────┐    ┌────────┐  │
└─→ │ Train    │ →  │ Score     │ →  │ Select │ ─┘
    │ Model    │    │ Unlabeled │    │ Top-K  │
    └──────────┘    └───────────┘    └────────┘
                                          │
                                          ↓
                                    ┌──────────┐
                                    │ Label    │
                                    │ Selected │
                                    └──────────┘

python

from src.active_learning import entropy_sampling

# After training your model on the current labeled set:
probs = model.predict_proba(X_unlabeled)
result = entropy_sampling(probs.tolist(), budget=100)

# Queue the selected samples for labeling
selected_tasks = [all_tasks[i] for i in result.indices]
engine.add_tasks(selected_tasks)

Handling Schema Changes

When you need to add/rename/merge categories mid-project:

python

from src.schema import SchemaMigration, MigrationStep, MigrationAction

migration = SchemaMigration(
    from_version="1.0",
    to_version="2.0",
    steps=[
        MigrationStep(
            action=MigrationAction.ADD_CATEGORY,
            params={"name": "neutral"},
            reason="Annotators struggled with ambiguous reviews",
        ),
    ],
)

# Apply to existing labels
migrated_labels, n_changed = migration.apply_to_labels(existing_labels)

Common Pitfalls

Problem	Symptom	Solution
Low agreement	κ < 0.4	Rewrite category descriptions with more examples
Annotator fatigue	Speed increases, accuracy drops per batch	Limit batch size to 50 items
Class imbalance	One label dominates > 80%	Use stratified sampling for batches
Schema creep	Too many categories	Merge rare categories, split only when needed
Gold item bias	Annotators recognise gold items	Rotate gold set regularly

Metrics to Track

1. Inter-annotator agreement (Cohen's κ / Fleiss' κ): target ≥ 0.6

2. Adjudication rate: should decrease over time as guidelines improve

3. Label velocity: items labeled per annotator per hour

4. Active learning efficiency: accuracy gain per labeled sample vs. random

5. Distribution drift: chi-squared p-value between weekly batches

Part of ML Engineer Toolkit

Chapter 2

Quality Assurance Guide for Data Labeling

Why Label Quality Matters

Bad labels are worse than no labels. A model trained on noisy annotations learns the noise — and you won't know until it fails in production. This guide covers the systematic approach to preventing, detecting, and fixing label quality issues.

The Three Pillars of Label QA

                    ┌─────────────────────┐
                    │   LABEL QUALITY     │
                    └─────────┬───────────┘
            ┌─────────────────┼─────────────────┐
            │                 │                 │
     ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐
     │  PREVENTION │  │  DETECTION  │  │  CORRECTION │
     │             │  │             │  │             │
     │ • Schema    │  │ • Agreement │  │ • Adjudicate│
     │ • Guidelines│  │ • Audits    │  │ • Re-label  │
     │ • Training  │  │ • Drift     │  │ • Retrain   │
     └─────────────┘  └─────────────┘  └─────────────┘

Pillar 1: Prevention

Clear Schema Design

The #1 cause of bad labels is ambiguous category definitions. Follow these rules:

1. Every category needs positive AND negative examples

2. Edge cases must be documented — "if unsure between X and Y, choose X because..."

3. Descriptions should be testable — can you verify whether a description applies?

Bad definition:

"positive: the review is positive"

Good definition:

"positive: the review expresses satisfaction, approval, or recommendation.
The reviewer would likely purchase again or recommend to others.
Example: 'Great quality, works exactly as described.'
NOT positive: 'It's fine I guess' (→ neutral)"

Annotator Training

Before starting production labeling:

1. Calibration round: All annotators label the same 20 items

2. Discussion: Review disagreements as a group

3. Guideline update: Incorporate learnings into the schema

4. Qualification test: Annotators must hit 85% on gold items

Golden Items

Mix pre-labeled items into every batch as quality checks:

python

# In your batch creation:
golden_fraction = 0.10  # 10% of each batch is gold
golden_items = sample_from_gold_set(batch_size * golden_fraction)

Key rules for golden items:

Rotate them frequently (annotators shouldn't memorise them)
Include all difficulty levels (not just easy ones)
Update the gold set when schema changes

Pillar 2: Detection

Inter-Annotator Agreement

Run agreement metrics after every batch:

python

from src.agreement import cohens_kappa, fleiss_kappa

# Two-rater agreement
result = cohens_kappa(rater_a_labels, rater_b_labels)
print(f"Cohen's κ = {result.kappa:.3f} ({result.strength})")

# N-rater agreement
result = fleiss_kappa([rater1, rater2, rater3])
print(f"Fleiss' κ = {result.kappa:.3f} ({result.strength})")

Agreement targets by task type:

Task Type	Target κ	Action if below
Binary sentiment	≥ 0.70	Rewrite guidelines
Multi-class (3-5 labels)	≥ 0.60	Add examples, recalibrate
Subjective rating	≥ 0.50	Consider merging categories
NER spans	≥ 0.65	Tighten span boundary rules

Annotator Scorecards

Track per-annotator quality continuously:

python

from src.quality_audit import build_scorecards

scorecards = build_scorecards(all_annotations)
for card in scorecards:
    if card.flagged:
        print(f"⚠ {card.summary()}")

Red flags:

Agreement rate below 70% → possible misunderstanding of guidelines
Very short duration → possible random clicking
High bias score → annotator systematically over-uses one category
Sudden drop in metrics → fatigue or guideline confusion after schema change

Suspect Label Detection

Find specific labels that are likely wrong:

python

from src.quality_audit import find_suspect_labels

suspects = find_suspect_labels(annotations, min_confidence=0.5)
for s in suspects[:10]:
    print(s.summary())

The confidence score combines:

Item agreement: How strongly do other raters disagree?
Annotator reliability: Is this annotator generally unreliable?

High confidence = high item agreement + low annotator reliability = almost certainly wrong.

Distribution Drift

Label distributions should be stable over time. If they shift, either:

The data changed (legitimate drift — update your model)
The annotators changed behavior (problem — investigate)

python

from src.quality_audit import LabelDistributionTracker

tracker = LabelDistributionTracker()
tracker.add_batch("week_1", week1_labels)
tracker.add_batch("week_2", week2_labels)
result = tracker.test_drift("week_1", "week_2")

if result.drifted:
    print(f"Distribution shifted! χ²={result.chi_squared:.2f}")

Pillar 3: Correction

Adjudication Workflow

When consensus fails:

1. Show the item + all submitted labels to a domain expert

2. Expert decides the correct label (may differ from all annotators)

3. Record the reasoning for future training

python

from src.workflow import WorkflowEngine

engine = WorkflowEngine(...)
for task in engine.tasks_needing_adjudication:
    # Present to expert with context
    engine.adjudicate(task.task_id, expert_label, "expert_01")

Handling Flagged Annotators

Decision tree for annotators with low agreement:

Agreement < 70%?
├── YES → Check golden item accuracy
│   ├── Gold accuracy < 80% → Re-train or remove annotator
│   └── Gold accuracy ≥ 80% → Possible guideline issue (investigate)
└── NO → Annotator is performing normally

Relabeling Strategy

When you find systematic errors:

1. Don't re-label everything — only items from flagged annotators

2. Prioritise high-value items — those used in validation/test sets

3. Use the suspect list — focus on highest-confidence suspects first

python

# Get the worst suspects
suspects = find_suspect_labels(annotations, min_confidence=0.6)

# Queue them for re-labeling by a different annotator
relabel_ids = [s.item_id for s in suspects[:100]]

Automated QA Pipeline

Set up a recurring quality check that runs after every batch:

python

def post_batch_qa(batch_annotations):
    """Run after every labeling batch completes."""

    # 1. Check agreement
    scorecards = build_scorecards(batch_annotations)
    flagged = [s for s in scorecards if s.flagged]
    if flagged:
        alert(f"{len(flagged)} annotators flagged")

    # 2. Check for suspects
    suspects = find_suspect_labels(batch_annotations, min_confidence=0.5)
    if len(suspects) > 10:
        alert(f"{len(suspects)} suspect labels found")

    # 3. Check distribution drift (compare to previous batch)
    tracker = LabelDistributionTracker()
    tracker.add_batch("previous", previous_labels)
    tracker.add_batch("current", [a.label for a in batch_annotations])
    drift = tracker.test_drift("previous", "current")
    if drift.drifted:
        alert("Label distribution drift detected!")

    # 4. Generate report
    report = generate_audit_report(batch_annotations)
    save_report(report, batch_id)

Quality Metrics Dashboard

Track these metrics over time:

Metric	Formula	Target	Frequency
Cohen's κ (pairwise)	(P_o - P_e) / (1 - P_e)	≥ 0.65	Per batch
Fleiss' κ (multi-rater)	(P̄ - P̄_e) / (1 - P̄_e)	≥ 0.60	Per batch
Adjudication rate	disputed / total tasks	≤ 15%	Per batch
Gold accuracy	correct_gold / total_gold	≥ 85%	Per annotator
Label velocity	items / hour / annotator	Stable	Weekly
Distribution drift	χ² p-value	> 0.05	Weekly

When to Stop Labeling

Your dataset is "done" when:

1. Agreement is stable — κ hasn't improved for 3+ batches

2. Model performance plateaus — more labels don't improve val metrics

3. Active learning scores are low — remaining unlabeled items are easy

4. No suspects — suspect detection returns zero or near-zero

At that point, do a final audit, export the approved labels, and move to model training.

Part of ML Engineer Toolkit

Data Labeling Pipeline v1.0.0 — Free Preview

Contents

Annotation Workflow Guide

Overview

The Labeling Lifecycle

Step-by-Step Setup

1. Define Your Schema

2. Configure the Workflow

3. Prepare Your Data

4. Create Batches and Assign

5. Collect Labels

6. Resolve Consensus

7. Adjudicate Disagreements

8. Run Quality Audit

9. Export Training Data

Active Learning Loop

Handling Schema Changes

Common Pitfalls

Metrics to Track

Quality Assurance Guide for Data Labeling

Why Label Quality Matters

The Three Pillars of Label QA

Pillar 1: Prevention

Clear Schema Design

Annotator Training

Golden Items

Pillar 2: Detection

Inter-Annotator Agreement

Annotator Scorecards

Suspect Label Detection

Distribution Drift

Pillar 3: Correction

Adjudication Workflow

Handling Flagged Annotators

Relabeling Strategy

Automated QA Pipeline

Quality Metrics Dashboard

When to Stop Labeling