← Master Index
Vol. 09 Module 9.1 Lecture

Sentence Segmentation

NLP Basics

How This Lesson Fits the Module & Volume

Tokenization splits text into tokens; many pipelines also need document text split into sentences. Sentence segmentation (sentence boundary detection) feeds sentence-level classifiers, translation, summarization chunking, and linguistic tools like POS tagging and dependency parsing.

Naive “split on periods” fails on abbreviations, decimals, and ellipses—exactly the mess left in a real corpus after light cleaning.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define sentence segmentation and why period-splitting is insufficient.
  • Identify hard cases: abbreviations, quotes, lists, and informal chat.
  • Segment sentences with NLTK and spaCy.
  • Explain when to segment before vs after tokenization.
  • Use sentence boundaries to chunk long documents for models with length limits.
  • Evaluate segmentation errors’ impact on downstream NLP.
Definition

Sentence segmentation is the task of locating sentence boundaries in running text and splitting a document into a sequence of sentence strings (or spans). It is also called sentence boundary detection (SBD).

Why Not Split on “.”?

TextNaive split riskCorrect intuition
Dr. Smith arrived.Breaks after DrAbbreviation, not end
Worth $3.50 today.Breaks on decimalNumber, not end
Wow!!! Really?Misses multi-punctTwo sentences / clauses
He said, “Go.” Then left.Quote/period interactionBoundary after quote

Practical Segmentation

import nltk import spacy from nltk.tokenize import sent_tokenize nltk.download("punkt_tab", quiet=True) text = ( "Dr. Lee met U.S. officials in Wash. D.C. " "Revenue rose 3.5%. Was that expected? Yes!" ) for i, s in enumerate(sent_tokenize(text), 1): print(i, s) nlp = spacy.load("en_core_web_sm") doc = nlp(text) print([sent.text for sent in doc.sents])

Where Segmentation Sits in the Pipeline

Document-level tasks

  • Topic classification.
  • Often skip sentence splits.
  • Use full cleaned text.

Sentence-level tasks

  • Sentiment per sentence.
  • Translation / NLI.
  • Segment first, then tokenize.

Long-context chunking

  • Pack whole sentences into windows.
  • Avoid cutting mid-sentence.
  • Better RAG / summarization chunks.

Rule / ML Segmenters

  • spaCy & NLTK handle many abbreviations.
  • Fast enough for batch jobs.
  • Good default for English prose.

Still Hard

  • Legal citations and lists.
  • Tweets / chat without punctuation.
  • Code-mixed and OCR text.
Common Misconception

“Sentence segmentation is just tokenization with bigger chunks.” Tokenization decides word/subword units; segmentation decides clause/sentence spans. You often need both: sentences for linguistics and document structure, tokens for model input.

Knowledge Check

  1. Short Answer: What is sentence segmentation? Answer: Locating sentence boundaries and splitting text into sentence units.
  2. True/False: Splitting on every period is reliable for English. Answer: False—abbreviations and decimals break that heuristic.
  3. Multiple Choice: Dr. Smith left. is hard because: (a) UTF-8, (b) abbreviation period, (c) stemming. Answer: (b).
  4. Short Answer: Name two libraries that provide sentence segmentation. Answer: NLTK (sent_tokenize) and spaCy (doc.sents).
  5. True/False: Document classification always requires sentence splits. Answer: False.
  6. Multiple Choice: Packing retrieval chunks on sentence boundaries helps: (a) avoid cutting mid-thought, (b) train GRUs faster only, (c) remove NER. Answer: (a).
  7. Short Answer: Why is chat text hard to segment? Answer: Missing punctuation, fragments, and informal line breaks.
  8. True/False: spaCy can provide both sentences and tokens in one Doc. Answer: True.
  9. Multiple Choice: SBD stands for: (a) subword byte decoding, (b) sentence boundary detection, (c) sparse bag dump. Answer: (b).
  10. Short Answer: What filtering step often follows tokenization for classical NLP? Answer: Stop-word removal (next lecture).

Key Takeaways

  • Sentence segmentation finds boundaries; period-splitting is not enough.
  • Use spaCy/NLTK (or specialized SBD) for production prose.
  • Segment when tasks or chunking need sentence units.
  • Errors cascade into POS, parsing, and evaluation metrics.
  • Next, Stop Words covers high-frequency tokens often filtered in classical pipelines.
Trainer’s Guide

Hands-on idea: Collect ten hard sentences (abbreviations, decimals, quotes) and score student regex splitters vs spaCy.

Discussion prompt: For RAG over PDFs, should chunking prefer sentences, paragraphs, or fixed tokens?

Recap: Reliable sentence boundaries unlock sentence-level NLP and smarter chunking. Continue with Stop Words.