OCR

How This Lesson Fits the Module & Volume

Vision can label a scene; OCR (optical character recognition) extracts the writing inside pixels—receipts, IDs, whiteboards, UI screenshots, scanned PDFs. It is a vision capability with a language output, sitting between CNNs/ViTs (Vol. 07 / Vol. 10) and NLP (Vol. 09 tokenization). Agents doing document work should prefer OCR (or a digital text layer) over a vague image caption.

OCR is not image generation (16.3) and not STT (16.2). It often runs inside video understanding when credits or burned-in subtitles matter.

Learning Objectives

By the end of this lesson, students should be able to:

Define OCR vs captioning vs general VQA.
Describe detect-then-recognize pipelines (boxes → text).
Choose digital PDF parse vs OCR vs multimodal VLM.
Run a practical OCR call in Python and post-process text.
List failure modes: skew, handwriting, low res, tables, languages.
Place OCR next to captioning in an agent toolbox.

Definition

OCR converts images of characters into machine-readable text (and usually bounding boxes, reading order, and optionally key–value structure). Handwriting recognition is a harder OCR variant. Document AI adds layout: tables, forms, signatures.

OCR vs Neighbor Capabilities

Task	Question it answers	Output
OCR	What characters are printed/written here?	Text + boxes
Captioning	What is the scene about?	One descriptive sentence
VQA	Answer a question about the image	Free text (may skip exact strings)
CLIP retrieval	Which image matches this phrase?	Similarity scores
Digital PDF parse	What is already in the text layer?	Exact Unicode (no vision)

Classic Pipeline

1. Detect

Find text regions / lines / words
CNN/ViT detectors (Vol. 07 YOLO family)
Deskew / denoise first

2. Recognize

Crop → character/word model
CTC or transformer decoder
Language model rescoring

3. Structure

Reading order, tables, KV pairs
Feed Vol. 09 tokenization / RAG
Agent tools consume JSON, not pixels

Practical Python OCR

# pip install rapidocr-onnxruntime pillow
from pathlib import Path
from rapidocr_onnxruntime import RapidOCR

ocr = RapidOCR()

def ocr_image(path: str) -> dict:
    result, _elapse = ocr(path)
    # result: list of [box, text, confidence]
    lines = []
    for item in result or []:
        box, text, conf = item[0], item[1], float(item[2])
        lines.append({"text": text, "conf": round(conf, 3), "box": box})
    full_text = "\n".join(x["text"] for x in lines)
    return {"text": full_text, "lines": lines, "source": Path(path).name}

def agent_observe_document(path: str) -> str:
    data = ocr_image(path)
    low = [ln for ln in data["lines"] if ln["conf"] < 0.6]
    warning = f"\n[low-confidence spans: {len(low)}]" if low else ""
    return data["text"] + warning
# Next: chunk this text for Vol. 14 RAG; do not embed the JPEG instead.

When to OCR vs When Not To

Scans, photos of paper, screenshots of UI text
Need exact strings (invoice totals, IDs)
Layout / tables for downstream extraction

Skip OCR

Native PDF/HTML already has text
You only need “this is a beach” (caption)
Illegible / adversarial images without human review

Common Misconception

“A multimodal LLM caption is OCR.” Captions paraphrase; OCR must reproduce characters. Asking GPT “what does this receipt say?” without boxes or a dedicated OCR pass is how totals get hallucinated. Use OCR for strings, captioning for scenes, VQA for questions—and verify numbers.

Knowledge Check

Short Answer: What does OCR output that captioning usually does not? Answer: Exact character strings (and typically bounding boxes).
True/False: Always OCR a digitally born PDF before RAG. Answer: False—extract the text layer first.
Multiple Choice: Detect-then-recognize means: (a) find text regions then read them, (b) generate a new image, (c) clone a voice. Answer: (a).
Short Answer: Which volumes supply the visual backbones OCR often uses? Answer: Vol. 07 CNNs and/or Vol. 10 ViTs.
True/False: Low OCR confidence should be hidden from the agent. Answer: False—surface it so humans/tools can verify.
Multiple Choice: Invoice total extraction is primarily: (a) OCR + parsing, (b) TTS, (c) video generation. Answer: (a).
Short Answer: Name two OCR failure modes. Answer: Skew, blur, handwriting, rare fonts, tables, language mismatch (any two).
True/False: OCR is catalogued as an image generator in 16.3. Answer: False—OCR is an understand capability in 16.1.
Multiple Choice: Burned-in video subtitles are closest to: (a) OCR, (b) voice cloning, (c) k-means. Answer: (a).
Short Answer: After OCR, which volume’s RAG pipeline usually consumes the text? Answer: Volume 14 (chunk / embed / retrieve).

Key Takeaways

OCR reads characters in pixels; captions describe scenes.
Detect → recognize → structure; then tokenize like any Vol. 09 text.
Prefer digital text layers; OCR scans and screenshots.
Never treat a VLM paraphrase as an invoice ground truth.
Next: Image generation (pixels out, not text out).

Trainer’s Guide

Lab: Same receipt photo: (1) OCR dump, (2) GPT caption, (3) GPT “read the total.” Compare strings. Students must flag hallucinated totals.

Extension: Photograph a whiteboard, OCR, then feed Vol. 14 RAG—close the multimodal → knowledge loop.

Recap: OCR is vision→text for written characters. Continue with Image generation.