Text Cleaning

How This Lesson Fits the Module & Volume

After you define a corpus, the text is rarely model-ready. HTML tags, broken encodings, copy-pasted signatures, and emoji noise inflate vocabulary and confuse tokenizers.

Text cleaning is the deterministic preprocessing step that normalizes surface form before tokenization, stop-word filtering, or embedding. Done well, it reduces noise without destroying meaning; done badly, it erases the signals your classifier needs (e.g., stripping all punctuation from “not good” vs “good”).

Learning Objectives

By the end of this lesson, students should be able to:

List common noise sources in real-world text corpora.
Apply Unicode normalization, casing policy, and whitespace cleanup with Python.
Use regular expressions to remove or rewrite HTML, URLs, and boilerplate patterns.
Decide which cleaning steps help classical NLP vs modern transformer pipelines.
Preserve a raw copy and version the cleaning pipeline for reproducibility.
Avoid over-cleaning that removes task-critical cues.

Definition

Text cleaning is the set of rule-based (and sometimes lightly statistical) transforms that convert raw document strings into a consistent textual form suitable for tokenization and modeling—without yet assigning linguistic labels like POS or NER.

What Usually Needs Cleaning

Noise	Example	Typical action
Encoding issues	`cafÃ©` instead of `café`	Decode/fix UTF-8; NFC normalize
Markup	`<br>Please help</br>`	Strip tags; keep text
URLs / emails	`https://ex.com/a?x=1`	Remove or replace with `<URL>`
Boilerplate	“Sent from my iPhone”	Pattern removal
Whitespace	Tabs, double spaces, NBSP	Collapse / strip
Case	`ERROR` vs `error`	Lowercase if task allows

Cleaning vs Later Pipeline Stages

String-level, mostly regex/rules.
No linguistic analysis yet.
Goal: consistent surface form.

Tokenization

Splits into tokens.
Depends on clean boundaries.
Covered next lectures.

Normalization (linguistic)

Stemming / lemmatization.
Changes word form, not markup.
Optional for neural NLP.

A Practical Cleaning Function

Keep steps explicit and testable. For many chat/ticket pipelines this is enough as a first pass:

import re
import unicodedata
from html import unescape

URL_RE = re.compile(r"https?://\S+|www\.\S+", re.IGNORECASE)
EMAIL_RE = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
TAG_RE = re.compile(r"<[^>]+>")
SPACE_RE = re.compile(r"\s+")

def clean_text(text: str, lowercase: bool = True) -> str:
    text = unescape(text)
    text = unicodedata.normalize("NFC", text)
    text = TAG_RE.sub(" ", text)
    text = URL_RE.sub("  ", text)
    text = EMAIL_RE.sub("  ", text)
    text = text.replace("\u00a0", " ")
    if lowercase:
        text = text.lower()
    text = SPACE_RE.sub(" ", text).strip()
    return text

raw = "PLEASE visit https://help.example.com NOW!!! Thanks"
print(clean_text(raw))
# please visit  now!!! thanks

How Much Cleaning Do Modern Models Need?

Still Clean Aggressively When

Building bag-of-words / TF-IDF (Module 9.2).
Deduplicating near-identical tickets.
Removing HTML from scraped pages.

Clean Lightly When

Using pretrained transformers with their own tokenizer.
Case or emoji is a feature (sentiment, toxicity).
Legal text where punctuation carries meaning.

Common Misconception

“Always lowercase and strip all punctuation.” That heuristic from early bag-of-words tutorials can destroy named entities (US vs us), product codes, and negation cues. Choose a casing and punctuation policy per task, and keep the raw text immutable.

Engineering Checklist

Store raw_text and clean_text as separate columns.
Unit-test regexes on a golden set of ugly examples.
Log cleaning version (git hash / function name) with each experiment.
Never silently drop documents—count empties after cleaning.

Knowledge Check

Short Answer: What is text cleaning in NLP? Answer: Rule-based transforms that normalize raw strings before tokenization/modeling.
True/False: Text cleaning is the same as lemmatization. Answer: False—lemmatization is linguistic normalization of word forms.
Multiple Choice: Replacing URLs with a placeholder token is useful because: (a) it trains better GPUs, (b) it reduces unique rare strings, (c) it does POS tagging. Answer: (b).
Short Answer: Why keep a raw copy of each document? Answer: So you can re-clean, audit, or change policy without losing original evidence.
True/False: Transformer tokenizers eliminate the need to strip HTML. Answer: False—markup still pollutes context; clean structure first.
Multiple Choice: Unicode NFC normalization helps with: (a) batch size, (b) equivalent characters with different byte sequences, (c) dependency arcs. Answer: (b).
Short Answer: Give one case where you should not lowercase. Answer: When case distinguishes entities/acronyms (e.g., US vs us) or is a sentiment cue.
True/False: Aggressive punctuation stripping is always safe for sentiment analysis. Answer: False—it can remove negation and emphasis cues.
Multiple Choice: Cleaning belongs: (a) after embedding lookup only, (b) before tokenization in the pipeline, (c) only inside the GRU. Answer: (b).
Short Answer: Name two noise types common in email corpora. Answer: Any two of: signatures, HTML, quoted replies, URLs, encoding errors, boilerplate.

Key Takeaways

Cleaning normalizes surface form; it does not replace linguistic analysis.
Regex + Unicode hygiene handle most production mess before tokenization.
Match cleaning intensity to the model family (classical vs pretrained neural).
Version the pipeline and never overwrite the raw corpus.
Next, Token defines the atomic unit that tokenization will produce.

Trainer’s Guide

Hands-on idea: Give a page of HTML email and ask students to write clean_text tests that assert placeholders for URL/email and no remaining tags.

Discussion prompt: For toxicity detection, which artifacts (emoji, caps, repeated punctuation) should stay?

Recap: Clean text is consistent text—ready for tokens, not yet analyzed. Continue with Token.