Technical advances in document understanding

Summary of Technical advances in document understanding

by Practical AI LLC

49m•December 2, 2025
Listen to Episode

Overview of Technical advances in document understanding

This Practical AI episode (fully connected format with hosts Daniel Whitenack and Chris Benson) surveys the current state and practical options for automated document understanding — why it matters, the model families involved, how their pipelines differ, and when to pick each approach. The discussion emphasizes real-world trade-offs (accuracy, structure preservation, compute needs) and highlights recent innovations such as document-structure models (e.g., DocTR/DockLing toolkits), vision‑language models, and DeepSeek OCR’s multi-resolution approach. Practical recommendations center on choosing the right mix of tools for RAG systems, document conversion, compliance processing, and other business workflows.

Key topics covered

  • Why document processing still matters (ubiquitous, high business value, often annoying manual workflows).
  • Classical OCR pipeline: how it works, strengths, and limitations.
  • Document-structure models (DockLing and toolkits): what they output and why structure matters.
  • Vision‑language (multimodal) models: architecture and use cases.
  • DeepSeek OCR: multi-resolution tiling + global context approach and why it’s different.
  • Practical trade-offs: compute requirements, robustness to messy scans, and fit for RAG pipelines.
  • Use-case guidance and when to combine models.

Models & pipelines explained

Classical OCR

  • Input: image → preprocessing → detect text regions.
  • Per-region recognition: CNNs/LSTMs (or similar) output probabilities for characters/words.
  • Output: plain text (often needs post-correction and layout reconstruction).
  • Pros: lightweight (can run on CPU / small models), efficient for clean text scans.
  • Cons: fragile to layout, low-res scans, and complex documents (tables, multi-column, math).

Document-structure models (e.g., DockLing / toolkits)

  • Purpose: extract layout primitives and classify regions (title, heading, paragraph, table).
  • Output: structured representation (JSON, Markdown, HTML) — a tree-like layout description.
  • Typical pipeline: detect layout primitives → classify → (optionally) pass labeled regions to OCR for text extraction.
  • Best for: preserving order/structure (important for conversions or feeding downstream systems).
  • Compute: heavier than simple OCR but often runnable on commodity GPUs; smaller variants exist.

Vision‑language (multimodal) models

  • Input: image + optional text prompt; architecture fuses a vision transformer and a language transformer.
  • Output: token stream (text), like an LLM response conditioned on image + prompt.
  • Strengths: general multimodal reasoning (Q&A about images, image-aware generation).
  • Weakness: fixed-resolution inputs (most models), invisibility of explicit layout mapping (no explicit, interpretable link region→text).
  • Use cases: interactive image-aware assistants, image classification / reasoning, sometimes ad hoc document distillation.

DeepSeek OCR (multi-resolution)

  • Core idea: represent a document both as a global page image and as many high-resolution tiles (image tokens) so you don’t lose tiny details when downsampling.
  • Tiles preserve high-res features (small fonts, equations, diacritics, fine alignment), combined with a global view to preserve ordering/context.
  • Pros: better at preserving layout, fine-grained symbols, math, and subtle typography than fixed‑resolution multimodal models.
  • Cons: larger model, requires GPUs (current generation); likely to see size/efficiency improvements over time.

Practical use cases & recommendations

  • Simple, clean text scans: use classical OCR (Tesseract, PaddleOCR) — low compute, fast.
  • Documents with rich layout (tables, multi-column, figure captions) and need to preserve structure (or to reconstruct into different formats): use a document-structure model + OCR. Good for document conversions, complex ingestion pipelines.
  • Feeding documents into RAG (retrieval-augmented generation): document-structure models are highly recommended because they preserve ordering and chunk coherence; renderable output isn’t required — structured Markdown/JSON is ideal.
  • Conversational/multimodal interfaces (ask a document a question, show images + ask): use vision‑language models or multimodal LLMs.
  • High-fidelity extraction with tiny fonts, code snippets, equations, or annotations: consider DeepSeek-style multi-resolution approaches.
  • Resource constraints: prefer smaller OCR / document-structure models; larger vision-language and DeepSeek variants need GPUs.

Trade-offs and selection checklist

  • Accuracy vs. compute: heavier multimodal or multi-resolution models generally yield better fidelity but need GPU and more memory.
  • Structure vs. interpretability: document-structure outputs (trees/JSON) are interpretable and useful for downstream pipelines; multimodal outputs are less interpretable (text appears but region mapping is implicit).
  • End goal matters: rendering a faithful replica (e.g., converting Keynote → PowerPoint) requires more than labels — rendering logic is hard; if you only need content for search/RAG, structure + text is typically enough.
  • Pipeline composition: many practical systems combine models (structure model → OCR → cleaning → embed/chunk for search/RAG).

Notable insights & quotes (paraphrased)

  • “Document processing is not boring — there’s a lot of technical diversity and active innovation compared to the relatively uniform LLM landscape.”
  • “For RAG systems, preserving document structure often matters more than visual rendering; Doc-structure models are very useful here.”
  • DeepSeek’s approach highlights a key limitation of many vision models: fixed input resolution can throw away critical details; tiling + global context helps preserve them.

Actionable takeaways (what to try next)

  • If you manage a document ingestion pipeline: add a document-structure stage (DockLing or similar) before OCR to improve chunk coherence for search/RAG.
  • For RAG-based knowledge stores: parse documents into structured Markdown/JSON and chunk around logical sections rather than using a flattened OCR dump.
  • For documents with equations/handwriting/tiny fonts: evaluate multi-resolution approaches (DeepSeek-like) to see if extraction fidelity improves.
  • Prototype cost/latency: test OCR-only vs. structure+OCR vs. multimodal on representative documents; measure text fidelity, chunk quality (for retrieval), and compute cost.
  • Monitor the space: expect model sizes and runtime requirements to change — newer smaller/faster variants will appear.

Sponsors & episode context

  • Episode is a “fully connected” conversation (hosts only) recorded before U.S. Thanksgiving; sponsors mentioned include Shopify, Fabi, and Framer (sponsors were read during the episode).

Closing summary

Document understanding today is a diverse field with multiple complementary approaches. Choose classical OCR when you need speed and low compute; use document-structure models when preserving layout and order is important (especially for RAG); use multimodal vision-language models for interactive or open-ended image reasoning; and explore multi-resolution techniques (like DeepSeek OCR) where tiny or complex visual details matter. The practical wins come from composing these tools to match your business goal (search fidelity, conversion, end-user QA) rather than treating one model as a silver bullet.