KnowForge improves reading and locating answers in mixed-format documents
Locate, Judge, Read: Role-Specialized Tool Agents over a Knowledge-Point Index for Open-Domain Multimodal Document QA
Artificial IntelligenceComputation and LanguageInformation Retrieval
Summary
Answering questions from a large set of different documents can be tricky because it needs both finding the right section and understanding it well. The authors created KnowForge, a system that splits these tasks among specialized parts to search and read more effectively. This approach improves accuracy on various test collections containing text, images, and tables. The separate agents work together, deciding when to keep searching or start reading, making the overall system better at answering complex questions.
What this means in practice
- •For enterprise data teams: Improve question answering accuracy over diverse company documents by assigning specialized search and reading tasks to different AI agents.
- •For digital library operators: Enhance user query handling in large digital archives by deploying KnowForge's pipeline to locate and interpret information across text, tables, and images.
Authors
Wenlin Lin, Fanzhe Wei
Abstract
Question answering over large heterogeneous document collections requires first finding the right passages and then reading them, yet current retrieval-augmented agents give one model both kinds of tools and let it schedule itself. We present KnowForge, a pipeline separated by job: a hybrid-retrieval knowledge-point index; a Locator agent that can only search; a code-scheduled bridge judge that forces the missing second hop; a typed, size-bounded evidence pool that merges every search's ranking; a Deep Reader agent that can only read text, images and tables; and a synthesizer with a format checker. With a self-hosted Qwen3.6-35B-A3B (3B active), KnowForge reaches F1 47.5 on the full open-domain M3DocVQA (2,441 questions, 3,368 PDFs), +11.0 over the published M3DocRAG system and +7.1 over one-shot retrieve-and-answer on the same index and model, and 53.5 with DeepSeek-V4-Flash. On MultiHop-RAG it reaches 73.5 accuracy, above the strongest published RAG and GraphRAG results (71.2); on the single-document MMLongBench-Doc it lifts the same model from 34.8 to 47.0, level with GPT-4o reading page images (46.3). Per-job ablations on the full benchmarks attribute the gain: the Deep Reader is worth 3.8, 9.8 and 0 points on the three corpora, tracking how visual each is; merging both agents into one with all tools and twice the rounds loses 1.9 F1 because it keeps searching instead of reading; a round-budget sweep shows a step at four Locator rounds and a plateau beyond; query latency scales with model calls rather than corpus size. We serve the three indexed benchmarks as read-only knowledge bases (free registration) and release all prompts and the runner code: https://github.com/metask-ai/knowforge-bench