Data Labeling — Module 4.1

How This Lesson Fits the Module

Supervised ML needs labels—human-defined outputs for inputs. After Data Cleaning stabilizes rows and columns, labeling assigns the target variable: spam/ham, defect OK/NOK, intent class, toxicity score.

Label quality often caps model performance more than architecture choice. AI engineers design guidelines, measure agreement, and build feedback loops—not only train models on whatever CSV exists.

Learning Objectives

By the end of this lesson, students should be able to:

Distinguish labels, annotations, and metadata in supervised pipelines.
Write labeling guidelines with examples, edge cases, and decision trees.
Compute inter-annotator agreement (percent agreement, Cohen’s kappa).
Structure label tables joinable to feature rows by stable IDs.
Plan adjudication when annotators disagree.
Estimate labeling cost, throughput, and gold-set monitoring.

Introduction: Labels Are the Product

In many AI teams, the dataset with labels is the durable asset; model weights are disposable experiments. Labels encode business rules: what counts as fraud, harassment, or a successful handoff. Ambiguous guidelines produce noisy targets and unstable metrics.

Definition — Supervised Label

A label is the target output y paired with input x for supervised learning. For row id=42, the label might be a class (intent=refund), a score (toxicity=0.82), or structured JSON ({"entities": [...]}).

Labels require a labeling guideline: written rules annotators follow so that two experts assign the same class given the same evidence.

Labeling Workflow

Stage	Owner	Deliverable
Task definition	PM + ML lead	Label schema, success metric
Guideline draft	Domain expert	Rules + positive/negative examples
Pilot	Annotators	50–200 double-labeled items
Agreement review	ML engineer	Kappa / disagreement report
Scale-out	Labeling ops	Production label table
Monitor	ML engineer	Gold questions, drift alerts

Guidelines That Annotators Can Follow

Strong guidelines include: task summary, class definitions, when to abstain, borderline examples, and a changelog. Link guidelines in the labeling UI so annotators do not rely on tribal knowledge.

Good Guideline Properties

Mutually exclusive classes where possible
Worked examples per class (3+ each)
Explicit precedence for overlapping rules
Version number and effective date

Common Failures

“Use your judgment” without anchors
Too many fine-grained classes early
Guidelines that drift without re-labeling
No process for new edge cases

Storing Labels in Tabular Form

import pandas as pd

features = pd.read_parquet("data/clean/tickets.parquet")
labels = pd.read_csv("data/labels/ticket_intent_v3.csv")

# One row per item — adjudicated final label
labels = labels.loc[labels["is_final"]]
train = features.merge(labels[["ticket_id", "intent", "guideline_version"]], on="ticket_id", how="inner")

print(train["intent"].value_counts(normalize=True))

# Double-label export for agreement study
pairs = (
    raw_labels.groupby("ticket_id")["intent"]
    .apply(list)
    .reset_index()
)
pairs["agree"] = pairs["intent"].apply(lambda xs: len(set(xs)) == 1)
print("percent agreement:", pairs["agree"].mean())

Inter-Annotator Agreement

Definition — Cohen’s Kappa

Cohen’s kappa measures agreement between two annotators beyond chance. Values near 1 indicate strong agreement; near 0 suggests chance-level; negative values suggest systematic disagreement.

Rule of thumb for hard subjective tasks: kappa > 0.6 before scaling labeling; revisit guidelines if lower.

from sklearn.metrics import cohen_kappa_score

a1 = ["refund", "billing", "refund", "other"]
a2 = ["refund", "billing", "other", "other"]
print("kappa:", cohen_kappa_score(a1, a2))

Engineering Habit Track guideline_version on every label row. When guidelines change, either re-label affected items or partition training sets so metrics remain comparable.

Adjudication and Gold Sets

When annotators disagree, a senior reviewer picks the final label or sends the item back to clarify the guideline. Embed gold questions (pre-labeled items) in each batch to detect annotator drift and fatigue.

Common Misconception: “100% agreement is required before training.”

Reality: Some tasks are inherently subjective. Measure agreement, improve guidelines, adjudicate edge cases, and report irreducible noise in baseline metrics.

Common Misconception: “Labeling is separate from ML engineering.”

Reality: Schema design, active learning, and model-in-the-loop pre-labeling are core ML engineering skills that directly affect data quality and cost.

Active Learning Sketch

Train a rough model, score unlabeled pool, send uncertain items to humans first—maximizing information per dollar.

# Uncertainty sampling — lowest max class probability
import numpy as np

proba = model.predict_proba(X_pool)  # sklearn classifier
uncertainty = 1 - proba.max(axis=1)
priority_idx = np.argsort(-uncertainty)[:500]  # top 500 to label next

Knowledge Check

Short Answer: What is a labeling guideline? Answer: Documented rules and examples for assigning labels consistently.
True/False: Labels should include guideline version metadata. Answer: True.
Multiple Choice: Cohen’s kappa adjusts for: (a) class imbalance only, (b) chance agreement, (c) model loss, (d) API rate limits. Answer: (b).
Short Answer: Why double-label a pilot batch? Answer: Measure agreement before scaling expensive labeling.
True/False: Gold questions help detect annotator drift. Answer: True.
Multiple Choice: Join labels to features on: (a) random index, (b) stable business ID, (c) row order after shuffle, (d) filename. Answer: (b).
Short Answer: What is adjudication? Answer: Resolving disagreements with a final authoritative label.
True/False: Active learning always eliminates human labeling. Answer: False.
Multiple Choice: Spatial bounding boxes are covered in: (a) this lecture only, (b) Data Annotation, (c) ETL, (d) matplotlib. Answer: (b).
Short Answer: One metric besides kappa for agreement? Answer: Simple percent agreement (or Fleiss’ kappa for 3+ raters).

Key Takeaways

Supervised labels encode business rules—guidelines matter as much as models.
Pilot, measure agreement, adjudicate, then scale labeling.
Store labels with IDs and guideline versions joinable to clean features.
Use gold sets and active learning to control cost and quality.
Next: Data Annotation for structured spatial and token-level targets.

Trainer’s Guide

Mini-project: Teams write a one-page guideline for ticket intent (3 classes), double-label 30 items, compute kappa, and revise the guideline once.

Discussion: When is it ethical to use model pre-labels for human review? What disclosure do annotators need?

What’s Next Move from class labels to rich structures—boxes, spans, masks—in Data Annotation.