Chinese essay correction from images needs better AI grounding
From Images to Grounded Edits: A Real-World Benchmark for Chinese Grammatical Error Correction in Handwritten Essays
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Correcting grammar in handwritten Chinese essays is hard because the text is in images, not digital form. The authors created a new task where AI not only fixes grammar mistakes but also shows exactly where those mistakes are on the essay image. They built a benchmark with real student essay images and tested current AI models on it. The results show even top AI tools struggle without breaking the problem into parts, and pinpointing exact errors and their locations remains challenging. This work highlights the need for AIs that combine understanding of pictures and language more effectively.
What this means in practice
- •For education technology developers: Build AI tools that directly correct handwritten student essays by locating and fixing grammar errors on essay images.
- •For document processing engineers: Improve pipelines that integrate OCR and grammatical correction for documents with handwritten text by using multi-agent workflows for error localization.
Authors
Weiru Zhang, Yuzhong Hong, Junwei Bao, Lemao Liu, Dejing Dou
Abstract
Chinese Grammatical Error Correction (CGEC) is typically formulated as a text-to-text problem. While suitable for digital texts, this formulation falls short in real-world K-12 education, where AI-assisted corrections are expected directly on essay images, which are primarily handwritten and exhibit many irregularities. To bridge this gap, we introduce Grounded Chinese Grammatical Error Correction (G-CGEC), an image-to-grounded-edit task requiring systems to predict both corrections and their positions on essay images. We propose G-CGEC-Eval, a benchmark comprising real-world images of K-12 Chinese essays with fine-grained annotations. For evaluation, we adapt IoU-based metrics from object detection and propose a more tolerant alternative based on center-distance matching. Using this benchmark, we evaluate open-source and proprietary vision-language models under a direct single-pass setting, as well as settings with auxiliary components (OCR and character grounding) coordinated via pipelining or agentic harnessing. Results show that even top-tier proprietary models perform surprisingly poorly in the single-pass setting, while pipelining and agentic harnessing yield meaningful improvements. Notably, an error-type-based multi-agent workflow achieves the best performance, establishing it as the strongest baseline for future G-CGEC research. Through ablations and fine-grained analysis, we find the task remains far from solved, with linguistic error discovery and character-level grounding as the primary bottlenecks. This underscores the need for models that better integrate visual perception with linguistic reasoning and visual grounding.