LLM grading feedback lacks true diagnostic value in computer science tests
Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structured Assessment in Higher Education
Computers and Society
Summary
Large language models (LLMs) are used to grade and give feedback on student answers in higher education, but this study shows their detailed scores and comments do not always match real human evaluation. The sub-scores given by LLMs are very similar and repetitive, so they don’t give independent insights. Their feedback rarely identifies where students misunderstand the material, unlike human teachers, and the tone is always positive, no matter how good or bad the answer is. How well the LLMs grade depends on the subject type, working best on step-by-step procedural topics.
What this means in practice
- •For educational technology developers: Design automated grading tools that highlight which feedback aspects from LLMs need human review due to low diagnostic accuracy.
- •For instructional design teams: Focus manual assessment efforts on conceptual and diagnostic feedback during rubric development, especially where LLM feedback lacks severity modulation.
Tested on one dataset.
Authors
Xi Zhao, Xinyue Jiao, Zhen Xu
Abstract
As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines whether these outputs deliver what they appear to offer. Using the JorGPT dataset of 3,041 student responses to 50 open-ended computer science questions, scored by both human instructors and three commercial LLMs, we identify three systematic discrepancies between the apparent and actual quality of LLM-generated grading and feedback. The sub-dimension scores are highly correlated (r = 0.82-0.99, VIF up to 45), providing redundant rather than independent diagnostic information. The textual feedback rarely detects student misconceptions (5-7% vs. 15-31% for teachers), functioning as a coverage checklist rather than a diagnostic instrument. The feedback tone remains uniformly positive regardless of response quality, lacking the severity modulation observed in human feedback. Additionally, grading accuracy varies significantly by knowledge domain, with procedural topics most reliable. These findings provide empirically grounded guidance on which aspects of LLM-generated grading and feedback can be relied upon and which require continued human oversight.