Large language models automate practical judgment tests with feedback
Automating Constructive Assessment with Large Language Models: Toward Scalable and Repeated Evaluation of Practical Competence
Computers and Society
Summary
Practical judgment tests usually need experts to create problems and grade answers, which takes a lot of time and effort. The authors developed a way to use large language models to automatically create cases with errors, score student answers, and provide helpful feedback. Their system's scores matched human graders closely, and the feedback was rated just as useful. This could make practical skills testing faster, cheaper, and easier to repeat over time.
What this means in practice
- •For corporate training teams: Automatically generate and grade error-based case studies for employee skill evaluations with immediate, structured feedback.
- •For online learning platform developers: Integrate automated practical judgment assessments that provide instant, human-quality scoring and feedback to scale student evaluation.$Commercial implications: The paper enables building a scalable education product that reduces instructor workload and improves learner engagement through automated assessment.
Authors
Satoshi Takahashi, Atsushi Yoshikawa, Megumi Kose, Kenichi Suzuki, Chieko Inoue, Yumi Watanabe, Mari Sawada
Abstract
This study aimed to automate hierarchical diagnostic reasoning (HDR), a constructive method for evaluating practical judgment skills, by developing and testing an evaluation process using a large language model. HDR is a descriptive task that measures higher-order cognitive skills by requiring students to identify and explain errors in case-study-based problems. However, it requires expertise and effort to develop and evaluate. Hence, we proposed and empirically validated the automatic (1) generation of case problems containing errors aligned with educational intentions, (2) scoring of descriptive answers, and (3) generation of structured feedback based on incorrect answers, achieved solely through prompt design without fine-tuning. The internal consistency and construct validity of the generated problems were supported by the experimental score distribution and Cronbach's alpha (0.78). The agreement between automated and human ratings reached 100% under some conditions. The feedback was rated as being as convincing and useful as that from human instructors, demonstrating a practical framework for implementing HDR-based constructive assessment with reproducibility, immediacy, and low cost. The flexibility of large language models will also enable repeated and longitudinal assessments while maintaining structure, showing broad potential for application in educational settings.