Automated essay scoring improves with error-aware synthetic writing
Error-Supervised Synthetic Learner Writing for Automated Essay Scoring
Computation and Language
Summary
Automated essay scoring systems need lots of real essays to learn how to grade well, but writing these essays is slow and expensive. The paper shows that by teaching a computer to make synthetic essays that include realistic mistakes like those made by language learners, these systems can be trained better. The authors found that scoring systems trained on these improved synthetic essays performed closer to those trained on real essays when there was a fair amount of data. This means it's possible to reduce the need for lots of real essays without losing much grading quality.
What this means in practice
- •For language assessment developers: Create better training data for essay grading tools by generating synthetic essays with realistic learner errors to improve scoring accuracy.
- •For language learning software teams: Generate practice essays that mimic typical mistakes made by language learners to improve feedback and testing systems.
Authors
Duy Anh Nguyen
Abstract
Synthetic essays can help reduce dependence on human-written data in Automated Essay Scoring (AES). However, they often lack realistic errors, limiting their ability to represent authentic human writing, particularly when the target texts are intended to resemble those produced by language learners. In this study, we present a simple approach that introduces error supervision into synthetic essay generation. Specifically, we fine-tune an LLM generator on error-annotated texts of the kind commonly used in Grammatical Error Detection (GED). To assess the utility of the proposed approach, we fine-tune and evaluate AES scorers under three data conditions: authentic essays, synthetic essays generated conventionally, and synthetic essays generated using our proposed approach. The results show that in the larger-data settings, the proposed approach outperforms the conventional synthetic baseline in 11 out of 12 dataset-metric comparisons, with performance in some cases approaching that of models trained on authentic essays. Despite these gains, performance under extremely low-resource settings remains mixed, with advantages over the conventional baseline only becoming more apparent at 200 training essays, although not consistently across datasets. Qualitative and quantitative analyses further show that the proposed approach produces learner-like errors whose distributions broadly resemble those observed in authentic essays.