Corpus — Module 9.1

How This Lesson Fits the Module & Volume

Volume 08 closed with gated sequence models—LSTM and GRU—that consume ordered inputs over time. Those networks are ready for language, but language does not arrive as clean tensors. It arrives as messy collections of documents, tickets, chats, and web pages.

This lecture opens Volume 09 and Module 9.1 by defining the corpus: the curated body of text you will clean, tokenize, tag, and later embed. Everything ahead—text cleaning, tokenization, POS tagging, NER—operates on a corpus. Module 9.2 then turns that processed text into vectors via one-hot encoding, bag-of-words, and learned embeddings.

Learning Objectives

By the end of this lesson, students should be able to:

Define a corpus and distinguish it from a single document or a raw dump of files.
Describe common corpus types used in applied NLP (domain, parallel, annotated, streaming).
List the metadata and quality checks an AI engineer should record for a production corpus.
Load and inspect a small corpus with Python (plain files and Hugging Face datasets).
Explain how corpus design affects downstream tokenization, labeling, and model bias.
Connect sequence models from Volume 08 to the text data pipeline that feeds them.

Definition

A corpus (plural: corpora) is a structured collection of authentic language samples—documents, utterances, or sentences—assembled for linguistic analysis or machine learning. A useful corpus is more than a folder of files: it has a defined domain, sampling criteria, encoding, and often labels or metadata.

From Sequence Models to Language Data

A GRU expects a sequence of vectors. Before you can produce those vectors, you need a corpus, then a pipeline that turns characters into tokens and eventually into embeddings. Skipping corpus design is how teams train on duplicated tickets, mixed languages, or PII-heavy dumps and wonder why evaluation looks great while production fails.

Concept	What it is	Example
Document	One coherent text unit	A support ticket
Corpus	Collection of documents under shared criteria	All tickets from 2024, English, product X
Dataset split	Train / val / test partition of a corpus	80 / 10 / 10 by ticket ID
Annotation layer	Labels or spans over the text	Intent labels, NER entities

Types of Corpora You Will Meet

General / Reference

Broad coverage (news, web, books).
Good for pretrained models.
May mismatch your domain.

Domain / In-house

Support logs, contracts, EHR notes.
Highest ROI for production NLP.
Needs cleaning and access control.

Annotated

POS, NER, parse trees, intents.
Enables supervised training.
Costly; check inter-annotator agreement.

What to Record About a Corpus

Treat the corpus like a dataset card, not a mystery zip file:

Source & license — where text came from and whether you may train on it.
Language & dialect — ISO codes, code-switching, OCR vs native digital text.
Time range — language drifts; old product names and policies matter.
Sampling — random, stratified by class, or “whatever we had.”
PII / toxicity — redaction policy before models memorize sensitive strings.
Size stats — document count, token count (after you define tokens), class balance.

Inspecting a Corpus in Python

Start small: load files, count documents, peek at lengths. Later lectures will clean and tokenize; here the goal is inventory.

from pathlib import Path
from datasets import load_dataset

# Local folder of plain-text documents
docs = []
for path in Path("data/tickets").glob("*.txt"):
    docs.append({"id": path.stem, "text": path.read_text(encoding="utf-8")})

print(f"documents: {len(docs)}")
print(f"avg chars: {sum(len(d['text']) for d in docs) / max(len(docs), 1):.0f}")
print(docs[0]["text"][:200])

# Public corpus via Hugging Face Datasets (cached after first download)
ds = load_dataset("ag_news", split="train[:1000]")
print(ds)
print(ds[0])  # {'text': '...', 'label': 2}

Corpus Design Affects Everything Downstream

Good Corpus Habits

Split by document ID, not by sentence (avoids leakage).
Keep raw + cleaned versions versioned.
Document label guidelines before annotation.

Common Failures

Training and test share near-duplicate emails.
English-only assumptions on multilingual tickets.
No audit trail when sources change monthly.

Common Misconception

“Bigger corpus is always better.” Unfiltered web scrapes can drown your signal in spam, boilerplate, and duplicated boilerplate. For supervised tasks, a smaller, well-labeled, domain-matched corpus usually beats a huge noisy dump—especially before you reach representation learning in Module 9.2.

Bridge to the Rest of Module 9.1

1. Corpus

Define and inventory the text.

2. Clean

Normalize noise.

3. Tokenize

Split into tokens.

4. Analyze

POS, NER, parses—then embed in 9.2.

Knowledge Check

Short Answer: What is a corpus? Answer: A structured collection of authentic language samples assembled for analysis or ML.
True/False: A single PDF is already a corpus. Answer: False—a corpus is a collection under shared criteria; one document is just a document.
Multiple Choice: Splitting train/test by random sentences from the same emails often causes: (a) faster training, (b) data leakage, (c) better stemming. Answer: (b).
Short Answer: Name three metadata fields you should record for a production corpus. Answer: Any three of: source/license, language, time range, sampling method, PII policy, size stats.
True/False: Domain corpora are usually more valuable for production NLP than generic web text. Answer: True.
Multiple Choice: An annotated corpus includes: (a) only raw bytes, (b) labels or spans over text, (c) only embeddings. Answer: (b).
Short Answer: How does Volume 08 (GRU/LSTM) connect to this lecture? Answer: Sequence models need ordered inputs; the corpus is the language data that will be cleaned, tokenized, and turned into sequences.
True/False: Larger unfiltered scrapes always improve supervised classifiers. Answer: False—noise and duplication can hurt more than they help.
Multiple Choice: Hugging Face datasets is useful for: (a) GPU kernels, (b) loading and slicing public corpora, (c) dependency parsing only. Answer: (b).
Short Answer: What comes immediately after corpus design in Module 9.1? Answer: Text cleaning (then tokens/tokenization).

Key Takeaways

A corpus is a curated collection of language samples with criteria, encoding, and often metadata or labels.
Corpus design determines leakage risk, bias, and how well models transfer to production.
Inventory size, language, license, and PII before you invest in labeling or training.
Volume 08 gave you sequence models; Volume 09 starts by getting the text right.
Next, Text Cleaning turns raw corpus text into something tokenization can trust.

Trainer’s Guide

Hands-on idea: Give students a messy folder of emails and ask them to write a one-page corpus card (source, language, duplicates, PII, proposed train/test split).

Discussion prompt: Would you rather have 10k carefully labeled tickets or 10M scraped web pages for an intent classifier—and why?

Recap: NLP begins with the corpus you choose and document. Continue with Text Cleaning.