Large language models struggle to match teacher feedback adaptivity on student writing
Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing
Computation and LanguageArtificial Intelligence
Summary
This paper looks at how well big language models, like the ones behind chatbots, can give feedback on student writing like expert teachers do. The researchers compared types of feedback and how well the feedback changes depending on the student’s draft stage and performance. They found that while these models can cover many feedback types, they don’t match teachers in how they adapt their comments. The authors created a dataset to help improve this area in the future.
What this means in practice
- •For educational software developers: Improve writing assistance tools by using the dataset to align AI-generated feedback more closely with expert teacher practices.
- •For edtech content creators: Design AI tutoring content that better adapts feedback across student skill levels and draft progressions using insights from feedback adaptivity failures.
Authors
Norah Almousa, Shayan Peyghambari Oskoui, Raquel Coelho, Gayle Rogers, Xiang Lorraine Li, Diane Litman
Abstract
We investigate whether state-of-the-art large language models (LLMs) generate feedback that reflects the pedagogical practices of expert teachers in terms of feedback focus and adaptivity. Previous evaluation efforts have examined feedback characteristics, its impact on learning, and its target, yet the focus of feedback and its adaptivity remains largely overlooked. To bridge this gap, we adopt and refine Narciss's taxonomy into seven feedback focus types to annotate teacher and LLM-generated feedback across three university writing courses. We release FeedType, a benchmark containing annotated teacher and LLM feedback from six LLMs under three prompting strategies. We assess the coverage and distribution of feedback focus types, and examine whether LLMs adapt their feedback across draft stages and student performance levels as an expert instructor does. Our findings show that while most LLMs cover most feedback focus types, they fail to reflect teacher feedback distributions and show varying levels of adaptivity, with none matching the teachers' adaptive behavior. We believe FeedType will support future research on pedagogical alignment in LLM feedback generation.