Summary
Grading big computer science exams takes a lot of time and skilled people, so using AI graders is an attractive idea. The authors tested many versions of large language models (LLMs) to grade two different computer science exams and found some models performed better than human graders in agreement. However, how the AI was told to grade – especially strict instructions against partial credit – sometimes caused the AI to fail or refuse to grade. With light additional training on real graded examples, these models became more reliable and less sensitive to harsh grading instructions. The team also shared their data and tools to help others test or improve AI grading.
What this means in practice
- •For education technology companies: Develop AI grading systems for large-scale computer science exams matching human grader accuracy with less labor and cost.$Commercial implications: Enables scalable, cost-effective AI grading tools selling to universities and testing platforms, reducing human grader workload.
- •For online course platforms: Integrate fine-tuned language models to reliably grade coding exams with minimal setup and fewer grading errors.
Authors
Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh, Salman Khan, Naeemullah Khan
Abstract
One long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce; LLM graders are a tempting alternative. To show its pitfalls we grade a practical Computer Vision exam ($570$ dual-graded students) under $171$ configurations spanning closed and open-weights models; the best reaches mean absolute error $1.64/35$, below the $2.61/35$ two human graders achieve against each other. The catch is the prompt: a short ''strict grader'' preamble drives $14$ of $17$ open-weights models out of the graded band ($\text{MAE} \ge 8$), three stopping grading altogether. The damage traces to the preamble's two credit-withholding sentences, not to tone or model scale; one of them, ''never give partial credit'', alone makes two of three probed models stop grading. The closed flagships of three vendors shift calibration under it but stay in the band. In $162$ further configurations on a second, independent Machine Learning exam from another course ($1{,}038$ dual-graded students), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over-mark: the vulnerability replicates, but its direction is exam-specific. Light LoRA fine-tuning repairs it: one adapter on the two exams' pooled $\sim 3{,}900$ graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes ($\le 0.32$ MAE). We release the anonymised dataset, full ablation grid, and grading, fine-tuning and analysis pipelines.