SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework
2026-06-30 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors created SEFORA, a large public dataset that links college writing feedback from instructors with the related assignments and drafts, to help improve automated writing feedback. They also developed UniMatch, a new way to measure how well AI-generated feedback matches actual instructor comments by breaking feedback into pieces and comparing them carefully. Testing many AI setups showed that current models do not closely match instructor feedback and tend to get worse when generating more suggestions. This work highlights the challenges in making AI feedback as helpful and accurate as human feedback in real teaching.
writing feedbacklarge language modelsdatasetevaluation frameworksemantic similarityprecisionrecallF1 scoreautomated feedbacknatural language generation
Authors
Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou, Carolina Gustafson, Gayle Rogers, Raquel Coelho, Diane Litman, Xiang Lorraine Li
Abstract
Effective writing feedback is among the strongest drivers of student learning, yet producing it at scale is labor-intensive. LLMs offer a natural path to scaling writing support, but two gaps stand in the way: few public corpora capture how instructors actually deliver feedback in real classrooms, and no reliable method measures whether generated feedback aligns with what an instructor would write. We address both. SEFORA is a public corpus pairing instructor inline feedback with assignment prompts, rubrics, scores, and multi-draft revisions across various college writing genres, comprising 564 drafts and 8,240 instructor annotations. UniMatch is a reference-based evaluation framework for open-ended generation: it segments feedback into feedback units, scores their semantic correspondence under instructor-derived criteria, and aligns them via optimal matching to yield interpretable precision, recall, and F1. Across 74 experimental configurations spanning multiple LLMs, no setting exceeds 0.4 F1. UniMatch reveals that models struggle to identify the feedback instructors would prioritize, and performance degrades as models generate more.