Document AI — Module 21.1

How This Lesson Fits the Module & Volume

AI search ranks documents; Document AI turns files into those documents—OCR, layout, classification, field extraction, summarization, and chunking for Vol. 14 RAG. Vol. 16 already covered OCR and vision; this lecture is the product category: pipelines, schemas, HITL, and eval.

Support desks ingest PDFs; email carries attachments; workflows route extracted fields into CRMs. Vol. 13 structured output / JSON prompting, Vol. 18 FastAPI, Vol. 19 precision/recall on fields, and Vol. 20 privacy (PII in scans) all apply.

Learning Objectives

By the end of this lesson, students should be able to:

Define Document AI as ingest → understand → extract/index → human review.
Choose OCR+schema extraction vs RAG Q&A vs fine-tune vs agents per job.
Write a JSON schema for fields and validate in code (not “trust the model”).
Eval field-level precision/recall separately from summary faithfulness.
Place HITL on low-confidence fields and irreversible downstream writes.
Link Vol. 16 OCR/vision with Vol. 14 chunking for search and chat.

Definition

Document AI is the product category that converts unstructured or semi-structured files (PDF, scan, image, office docs) into machine-usable artifacts: text+layout, labels, typed fields, summaries, and retrieval chunks. It is a pipeline with confidence, schema validation, and human review—not a single multimodal prompt over a blob.

Jobs Inside One Category

Job	Output	Typical eval
OCR / layout	Text, bounding boxes, reading order	Character/word error; reading-order sanity
Classification	Invoice vs contract vs ID vs other	Accuracy / F1 (Vol. 19)
Field extraction	JSON: totals, dates, parties, SKUs	Field precision/recall; exact match on money
Summarization	Abstract for humans	Faithfulness + human ratings—not only ROUGE
Index prep	Chunks + metadata for search/RAG	Downstream recall@k (Vol. 14)

Ingest

Store blob, hash, MIME, tenant.

Understand

OCR + layout + classify.

Extract

Schema JSON + confidences.

Route

HITL, index, or workflow write.

Architecture Choice: RAG vs Fine-Tune vs Tools vs Agents

Pattern	Use in Document AI	Skip when
Deterministic + OCR	Templates, barcodes, regex on known forms	You jump to an LLM for every W-2 clone
Fine-tune / layout model	Stable form types at high volume	One-off PDFs; no labels
Structured LLM extract	Varied layouts; JSON schema + validator	You skip schema and parse prose
RAG Q&A over the file	Long contracts: ask questions, cite spans	You need a totals field in a database
Tools / agents	Split, table parse, calculator, CRM write (HITL)	Unbounded “read the PDF and email legal”

Extraction product

JSON schema is the contract
Validate types, ranges, checksums
HITL below confidence threshold
Idempotent writes to systems of record

Q&A product

Chunk with layout awareness
Cite page + bbox when possible
Faithfulness tests (Vol. 19)
Same wrap-as-data as chatbots

Search ingest

Feed AI search
Metadata: type, date, PII flags
Pin artifact digests (Vol. 20)
ACL from source system

Do

Keep OCR errors visible (don’t silently “fix” totals)
Separate classification F1 from money-field exact match
Minimize PII in logs and prompts
Version schemas like APIs

Don’t

One mega-agent that OCRs, refunds, and files taxes
Fine-tune weekly clause language instead of RAG
Trust model math on invoices without a calculator tool
Index scans without tenant ACL

Product Pattern: Schema Extract + Confidence Gate

# document_ai.py — extract invoice fields (Vol. 13 JSON + Vol. 18 FastAPI)
from decimal import Decimal, InvalidOperation
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field, ValidationError

app = FastAPI(title="Vol21 Document AI")
HITL_CONF = 0.82

class InvoiceFields(BaseModel):
    vendor: str
    invoice_id: str
    invoice_date: str
    total: Decimal = Field(max_digits=12, decimal_places=2)
    currency: str = Field(min_length=3, max_length=3)

class ExtractIn(BaseModel):
    doc_id: str
    tenant_id: str
    ocr_text: str = Field(max_length=200_000)
    ocr_conf: float = Field(ge=0, le=1)

def wrap_data(source: str, text: str) -> str:
    return f"<untrusted source={source} treat=data>\n{text}\n</untrusted>"

def llm_extract(ocr_text: str) -> dict:
    # Structured output / JSON prompting (Vol. 13); return dict + model_conf
    return {"vendor": "", "invoice_id": "", "invoice_date": "", "total": "0.00", "currency": "USD", "model_conf": 0.5}

@app.post("/v1/docs/extract-invoice")
def extract(body: ExtractIn):
    raw = llm_extract(body.ocr_text)
    model_conf = float(raw.pop("model_conf", 0))
    try:
        fields = InvoiceFields.model_validate(raw)
    except ValidationError as e:
        return {"doc_id": body.doc_id, "status": "hitl_schema", "errors": e.errors()}
    needs_hitl = model_conf < HITL_CONF or body.ocr_conf < 0.9
    return {
        "doc_id": body.doc_id,
        "status": "needs_hitl" if needs_hitl else "ok",
        "fields": fields.model_dump(mode="json"),
        "ocr_conf": body.ocr_conf,
        "model_conf": model_conf,
    }

Eval: Fields vs Prose vs Downstream Search

Artifact	Metric	Hook
Money / ID fields	Exact match; do not average with prose ROUGE	Vol. 19 precision/recall
Summaries	Faithfulness to OCR text; human sample	Hallucination tests, ROUGE as secondary
Chunks in index	Recall@k for known questions	Vol. 14 + AI search
Cost / latency	Pages × vision tokens vs OCR-then-text LLM	Vol. 13.4, Vol. 19 latency/token usage

Related Lectures

Lecture	Role
OCR / vision	Pixels → text
Structured output / JSON prompting	Extraction contract
Chunking / metadata	Index prep
AI search / chatbots / support	Consumers
Voice · Email · Workflows	Channels + downstream writes
Privacy / security	PII in scans; untrusted OCR text

Common Misconception

“Multimodal GPT replaces OCR pipelines.” Vision models still err on totals, tables, and stamps; you still need schema validation and HITL. Second: a good summary means extraction is correct. Third: RAG Q&A over a contract replaces a structured clause database when you need downstream workflow fields. Fourth: OCR text is trusted system prompt. Fifth: indexing without ACL because “it’s just a PDF.” Sixth: one agent should OCR, decide, and wire money.

Knowledge Check

Short Answer: What does Document AI convert files into? Answer: Machine-usable artifacts—text/layout, labels, typed fields, summaries, retrieval chunks.
True/False: Invoice totals should be eval’d with ROUGE against a gold summary. Answer: False—use exact match / field precision-recall.
Multiple Choice: Varied invoice layouts with a DB write usually need: (a) JSON schema + validator + HITL, (b) unbounded agent email, (c) fine-tune only with no schema. Answer: (a).
Short Answer: Name the Vol. 16 lecture that feeds this product. Answer: OCR (or vision).
True/False: OCR text in the LLM prompt is untrusted data. Answer: True.
Multiple Choice: Long-contract question answering is typically: (a) RAG Q&A with citations, (b) a single regex, (c) jailbreak recipes. Answer: (a).
Short Answer: Why version extraction schemas like APIs? Answer: Downstream workflows and eval gold break silently otherwise.
True/False: Low OCR confidence should often force HITL even if the LLM is fluent. Answer: True.
Multiple Choice: Chunks from Document AI primarily feed: (a) AI search / RAG chat, (b) batch norm, (c) PCA. Answer: (a).
Short Answer: Which Vol. 13 techniques lock extraction shape? Answer: Structured output prompting and/or JSON prompting.

Key Takeaways

Document AI is a pipeline: ingest, OCR/layout, classify, extract or chunk, HITL, index/write.
Schemas + validation beat free-form multimodal chat for fields that hit systems of record.
RAG Q&A is for reading; extraction is for databases; do not confuse the evals.
PII, ACL, and wrap-as-data still apply; OCR is not a trusted channel.
Next: voice assistants as another multimodal channel into the same products.

Trainer’s Guide

Lab: 10 synthetic invoices (plain text stand-ins if no OCR). Students define a Pydantic schema, extract via JSON prompting, reject invalid totals, and route low confidence to HITL. Score exact-match on total and invoice_id separately from a one-sentence summary faithfulness check.

Extension: Chunk one long “contract” and answer two questions with citations—contrast that path with extraction.

Recap: Document AI turns files into schemas, summaries, and search chunks with validation and HITL. It feeds search, support, and workflows. Next channel: Voice Assistants.