Lightweight vision language models extract text and structure from heritage documents
From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Many institutions struggle to digitize historical documents because existing tools often rely on costly and opaque commercial services. The authors studied eight smaller, open-source vision-language models to read text and organize it into structured data from images of old documents with complex layouts. They tested these models on real heritage collections and showed how tuning techniques improve accuracy without needing huge resources. Their work offers practical advice for institutions wanting private, cost-effective automated transcription and data extraction.
What this means in practice
- •For archival digitization teams: Build custom OCR tools to automatically transcribe and structure text from historical documents without relying on commercial APIs.
- •For software engineers in document management: Integrate lightweight vision-language models into existing workflows to convert document images into validated JSON data for downstream processing.
- •For legal tech companies: Use privacy-controlled vision-language models to extract structured data from scanned legal documents with domain-specific terminology.$Commercial implications: Enables selling compliant, private document processing services to law firms and regulatory bodies with sensitive data.
Authors
Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen
Abstract
While massive, closed-source Vision-Language Models (VLMs) set strong benchmarks for document understanding, their dependence on commercial APIs limits adoption in institutional archives due to data autonomy concerns, recurring costs, and the environmental footprint of hyperscale computing. This is especially acute in heritage digitization, where documents include historical handwriting, domain-specific terminology (e.g., jewelry, prehistory, architecture), and non-standard layouts requiring high-dimensional structured extraction. We present a comparative study of eight open-source lightweight VLMs (up to 7B parameters) for Optical Character Recognition (OCR)-to-structure across three university heritage collections. Given a document image, models must extract text and generate schema-compliant JSON, enabling automatic validation and downstream use. We evaluate models under a constraint-aware protocol across zero-shot, few-shot, and fine-tuning settings, measuring extraction fidelity and structured-output quality using Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS*), and mean Average Precision F1 (mAP-F1). Against a fine-tuning baseline, we further test the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and CLAHE), and (iii) multi-stage training. Finally, we analyze the trade-off between dataset-specific fine-tuning and a single multi-dataset checkpoint, where joint training enables one model to operate across collections but can shift performance between datasets. Overall, we show that carefully adapted VLMs with up to 7B parameters can provide a sustainable, private, high-performing alternative to manual transcription or commercial black-box systems, and we offer actionable guidance for heritage institutions seeking institution-controlled OCR-to-JSON extraction.