New reward system evaluates grammar corrections by final quality
Don't Count the Edits, Judge by the Outcome Alone: Reward-Based Evaluation for Grammatical Error Correction
Computation and LanguageArtificial Intelligence
Summary
Grammatical error correction tools usually check if a corrected sentence matches a reference version, which can unfairly judge other valid fixes. The authors created SURE, a new way to score corrections based on how well the output fixes the grammar, keeps the original meaning, and sounds natural, without relying too much on exact matches. SURE learns from examples that include both small fixes and complete rewrites and can identify where errors are fixed in the original sentence. Tests show SURE works well compared to existing methods, especially for more creative corrections.
What this means in practice
- •For grammatical correction software developers: Improve automatic evaluation of grammar correction tools by reliably scoring diverse valid sentence rewrites without relying on fixed reference texts.
- •For language technology service teams: Provide better quality assessment for grammar correction outputs in applications like writing assistants and educational platforms by using source-aware reward evaluation.
Authors
Hayeong Ryu, Sunhee Jo, Seunguk Yu, YoungBin Kim
Abstract
Grammatical error correction (GEC) evaluation has traditionally relied on reference or edit overlap, which can penalize valid rewrites that differ from gold corrections. Reference-free metrics reduce this dependence, but evaluating whether a fluent output is a valid correction of the source remains challenging. We propose SURE, a source-conditioned reward evaluator trained on within-source preferences spanning minimal-edit and rewrite-oriented corrections. SURE jointly learns an overall reward with criteria-level supervision for grammaticality, faithfulness, and fluency, together with span-level grounding for source-side error resolution. Experiments on SEEDA show that SURE performs competitively against strong baselines, with particular gains on rewrite-style corrections and more disentangled criteria-level diagnostics. Our code is available at https://github.com/hayeonggg/SURE.