Papers for

automated grading teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Large language models struggle to grade olympiad math proofs accurately

Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs

Abstract: Large Language Models (LLMs) have made remarkable progress on solving competition-level mathematics, yet their ability to verify natural-language mathematical proofs remains relatively underexplored. This gap is especially pronounced for Olympiad-level proofs, which require meticulous assessment of each claim in a long reasoning chain. Current proof verification thus primarily relies on expert inspection, which is costly to scale. In this paper, we present a systematic study of LLM-based verification in this setting. Specifically, we introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading, and conduct a broad evaluation of existing LLM verifiers across frontier models. We also propose TROJAN, an iterative self-critique pipeline for generating high-fidelity adversarial proofs at scale. Our comprehensive evaluation covers 20 models spanning seven metrics, two prompt styles, three inference methods, and three prompt templates, showing that state-of-the-art models still leave considerable room for improvement. To narrow this gap, we propose MABGrader, a Multi-Armed Bandit framework that reframes grading as arm selection over the discrete score space. MABGrader outperforms the SOTA ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%. Code and dataset are available at https://github.com/aziksh-ospanov/TROJAN.

Tue 27 OctArtificial Intelligence
The gist
Checking math proofs from difficult contests usually needs experts who carefully read each step, which is slow and expensive. The authors tested if large language models (LLMs) can help grade these proofs automatically. They made a new dataset of contest solutions graded by experts and tried many existing LLM graders. The results showed current models still make many mistakes. To improve this, the authors created a new system called MABGrader that uses advanced techniques to better match expert grading.
Open → 2610.36767v1

Multimodal verifier improves checking scientific images with explanations

SciGen-Verifier: A Multimodal Reasoner for Explainable Verification in Scientific Image Generation

Abstract: In realistic education, a solution is often expressed not only in words but in a drawing--a circuit, a geometric construction, a function plot--and a teacher must grade the drawing as carefully as the text. Recent advances in unified multimodal models have enabled scientific image generation, yet verifying the correctness of these specialized visual outputs remains a critical bottleneck: errors often arise from intricate domain knowledge, structural reasoning, and multi-step instruction rather than surface-level artifacts. Existing verifiers mainly target natural images and compress judgement into scalar scores, leaving scientific coverage and explainable feedback for error correction underexplored. To bridge this gap, we make three main contributions. (1) We construct SciGen-Verify, a benchmark dedicated to explainable verification of scientific image generation, spanning instruction following, multidisciplinary reasoning, and world knowledge domains. It contains a three-tier hierarchical protocol over the binary judgement, supporting explanation, and corrective editing instruction. (2) We develop SciGen-Verifier, a reasoning-driven multimodal verifier trained via cold-start supervised fine-tuning followed by a curriculum-based two-stage reinforcement learning pipeline. The rubric-guided process rewards first strengthen scientific reasoning exploration and outcome rewards subsequently align output with ground-truth annotation. (3) On SciGen-Verify, SciGen-Verifier achieves competitive performance against much larger proprietary models. It further serves as a practical online critic for iterative image rectification.

Sun 27 SeptComputer Vision and Pattern Recognition
The gist
School assignments often include drawings like circuits or graphs, but machines find it hard to check if these pictures are correct. The authors created a special test and a tool called SciGen-Verifier that can look at scientific images, understand instructions, and explain whether they are right or wrong. Their system learns step by step to reason carefully about the images and gives helpful feedback so mistakes can be fixed. It works well compared to bigger computer models and can help improve images by pointing out errors.
Open → 2609.33399v1

Typed classifier judges answers cheaper and faster than llms with similar mistakes

JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

Abstract: We ask whether Jev, a typed classifier that returns probabilities over permitted answers without generating text, can replace an LLM rubric judge. We compare it with three flash-tier LLM judges on nine panels drawn from seven benchmarks, giving every judge identical criterion texts. Jev's accuracy differs significantly from an LLM judge's in only 8 of 27 paired comparisons, ahead mostly on binary criteria and behind only on graded ones, and most of the other comparisons are inconclusive. Summed over the nine panels, the LLM judges, called once per criterion, cost 29 to 325 times as much as Jev and took 30 to 220 times as long. On graded criteria all four judges agree more with one another than with the labels and mostly assign lower levels than the raters. One of several observational accounts is that raters followed scale conventions our criterion texts omit. Jev's confidence ranks its own errors on most panels, which should make a cheap classifier the ideal first stage of a cascade that defers its uncertain verdicts to an LLM judge. Correlated errors undo that advantage. The LLM judges repeat nearly all of Jev's most confident errors, so a cascade replayed on the recorded verdicts lowers cost but gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 even with oracle thresholds.

Thu 24 SeptComputation and Language
The gist
The authors investigate whether a typed classifier called Jev can be used instead of large language models (LLMs) to judge answers based on rubrics. Jev is much cheaper and faster than LLM judges and performs similarly in accuracy, especially on simple yes/no criteria. However, both Jev and LLM judges tend to make similar mistakes, especially on more graded, complex criteria. This means using Jev first and then asking an LLM only on uncertain answers saves cost and time but improves accuracy only a little.
Open → 2609.29769v1