Vision language models locate and trace cursive handwriting strokes
inkGround: Benchmarking Fine-grained Visual Grounding Capabilities of Vision-Language Models for Cursive Handwriting in the Wild
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Handwriting is made by moving a pen in a particular order, but pictures only show the final ink marks. The paper introduces inkGround, a benchmark that tests if vision-language models (VLMs) can find specific words on photographed handwritten pages and trace the order of pen strokes that made the words. The authors created inkGround from real-world handwriting photos paired with predicted pen movement paths, checked by humans for accuracy. They tested several VLMs and found some, like Gemini-3.1-Pro, can closely match the correct stroke paths, helping machines better understand handwriting images.
What this means in practice
- •For software engineers: Develop handwriting recognition systems that accurately reconstruct pen stroke order from camera images of cursive handwriting.
- •For digital archivists: Use stroke trajectory recovery benchmarks to improve the digitization and analysis of historic handwritten documents.
Authors
Sidra Hanif
Abstract
Human handwriting is produced as an ordered sequence of pen movements, but an offline image keeps only the final ink. Handwriting trajectory recovery aims to restore the ordered stroke points and their pen-up/pen-down transitions. Existing image-trajectory pair datasets are recorded with digitizing tablets under controlled conditions, which limits writer coverage and stylistic diversity. Moreover, there is no benchmark that measures the visual grounding capabilities of general-purpose Vision-Language Models (VLMs) for the stroke trajectory recovery task. In this work, we introduce inkGround, a benchmark for stroke trajectory recovery from full-page, camera-captured handwriting. inkGround pairs 537 page images (~18.7k words, ~163k strokes) with reference trajectories produced by a pretrained offline-to-online model, filtered by OCR agreement, and verified by human raters. Because these references are pseudo-labels that represent one plausible pen trajectory, we also evaluate VLMs on the IAM-OnDB dataset, which provides recorded pen trajectories. We cast the task as text-conditioned point grounding: given a page image and a target word, the model must locate the word and output its ordered stroke points. Outputs are scored by trajectory fidelity (DTW, Chamfer distance), readability of the rendered strokes (Accurate Rate, Correct Rate), and style similarity. We benchmark four open-source and three frontier closed-source VLMs with a shared in-context prompt. Closed-source models, notably Gemini-3.1-Pro, produce legible trajectories close to the reference, and this ranking also holds on IAM-OnDB.