Area 02 · RAG · Concept 15 of 20

Document Parsing

Getting clean text, tables and structure out of PDFs, web pages and scans.

11–20 · Production levelPhase 2 · Weeks 7–10≈ 1.5 hours

The idea

Garbage in, garbage out. PDFs store positioned characters, not paragraphs; multi-column layouts, headers, footers and tables often come out scrambled. Web pages bring menus, cookie banners and ads. Scans need OCR.

Use the right tool per source: text-based PDFs with PyMuPDF or pypdf; complex layouts and tables with layout-aware parsers (Docling, Unstructured, LlamaParse); HTML with readability-style extraction; scans with OCR or a vision model. Convert everything to clean Markdown so headings and tables survive chunking.

Always eyeball the parsed output for a sample of files. Most retrieval bugs in real projects trace back to parsing.

Why it matters

Your RAG system can only find what parsing extracted correctly.

Words to know

  • OCROptical character recognition: reading text from images.
  • Layout analysisDetecting columns, headings and tables on a page.

Try it without code

Copy-paste test

  1. Open a two-column PDF and copy a page into a text editor.
  2. See where columns, tables and footers end up.
  3. That mess is what naive parsing gives you.

Use it in code

Extract page-tagged Markdown from a PDF with PyMuPDF4LLM.

python
# pip install pymupdf4llm
import pymupdf4llm

pages = pymupdf4llm.to_markdown("handbook.pdf", page_chunks=True)
for p in pages[:2]:
    print("--- page", p["metadata"]["page"], "---")
    print(p["text"][:500])
PRACTICE · TICK IT AFTER THIS

Parse 10 varied PDFs, save Markdown output, and write down which ones broke and how.

Common mistakes

  • Never inspecting parser output.
  • Dropping page numbers you need for citations.
  • Flattening tables into word soup.

Go deeper