DocPilot improves question answering on complex visual documents
DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding
Artificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Answering questions about long documents with images and text is hard because important information is spread out and in different forms. The authors created DocPilot, a system that can look through pages carefully and combine visual details, numbers, and words to find answers better. It uses small language models trained step-by-step to learn how to search and reason in these documents more like humans do. DocPilot works well on many tests and even smaller models perform strongly compared to much larger ones.
What this means in practice
- •For document processing teams: Use DocPilot to automate detailed extraction and reasoning over long and complex company reports or financial documents.
- •For legal tech developers: Implement DocPilot to improve answering questions about lengthy, multimodal legal documents requiring precise cross-referencing.
Authors
Zhuowen Liang, Xiaotian Lin, Yuyu Luo, Haixun Wang, Nan Tang
Abstract
Multimodal long-document question answering requires models to seek sparse evidence across pages and modalities, understand heterogeneous document content, and reason over the collected evidence. Even advanced vision-language models (VLMs) can overlook sparse evidence and struggle with cross-modal reasoning in a single pass. Existing document agents further introduce multi-step interaction, but some still rely on predefined workflows or coarse-grained actions, limiting the flexibility and granularity of document reasoning. We introduce DocPilot, an agentic document framework that supports expressive orchestration and makes complex document solving learnable by small language models (SLMs). DocPilot orchestrates evidence seeking through page-level navigation and fine-grained evidence localization, and integrates local visual inspection, code-based numerical reasoning, and cross-modal reasoning for multimodal understanding. To transfer these capabilities to small models, we propose curriculum-based agentic training that progresses from single-ability learning to adaptive orchestration through supervision from data-grounded reasoning traces and executable interaction trajectories. Across MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA, DocPilot achieves strong gains with frontier VLM backbones, while DocPilot-4B delivers competitive performance against substantially larger models, demonstrating strong effectiveness and generalization across diverse long-document reasoning settings. The code is available at https://github.com/HKUSTDial/DocPilot.