Accurate but Natural? Diagnosing Grammatical and Idiomatic Gaps in Japanese EFL Writing
2026-08-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how Japanese junior high students write in English, focusing on two things: making grammar mistakes versus not using natural-sounding phrases. They used a special method to separate simple grammar errors from awkward or unnatural expressions in their writing. Their results showed that some grammar parts are often wrong, while others are used too much or too little compared to native speakers. They proposed a way for teachers to tell if students need more practice fixing mistakes or just learning to use expressions more naturally. This helps teachers give better feedback tailored to each student's needs.
second language writinggrammatical accuracyidiomaticityautomated writing evaluationLLM correctionCEFR-J grammar extractoraccuracy gapsidiomatic gapsinstructional typology
Authors
Steve Woollaston, Brendan Flanagan, Hiroaki Ogata
Abstract
Second language writing research distinguishes grammatical accuracy from native-like idiomaticity, yet automated writing evaluation often conflates these dimensions. This study introduces a layered LLM-correction pipeline that isolates structural errors from unnaturalness by generating literal error corrections and idiomatic revisions for 3,830 English writing samples from 120 Japanese junior high school students. Applying the regex-based CEFR-J grammar extractor, we quantify two diagnostic measures: accuracy gaps (structures attempted but incorrectly produced) and idiomatic gaps (grammatically correct structures underused or overused relative to native norms). Results reveal distinct patterns: definite articles, third-person singular -s, and modals (would, could) exhibit significant accuracy difficulties, while -ing forms and hypothetical modals (would) show the largest idiomatic underuse, with simple present verbs, subject-verb-object patterns, and modal can conversely exhibiting the most pronounced overuse. A two-dimensional instructional typology maps error rates against idiomatic gaps, distinguishing accurate but overused grammar items from error-prone or avoided complex forms requiring targeted production practice. This framework advances pedagogical feedback by enabling teachers to diagnose whether learner difficulties arise from inaccurate execution, structural avoidance, or L1-mapped overreliance, supporting evidence-based interventions tailored to the specific needs of each learner.