Large language models struggle to grade olympiad math proofs accurately
Can LLMs Reliably Grade Olympiad Proofs? A Study of Mathematical Verification with LLMs
Artificial Intelligence
Summary
Checking math proofs from difficult contests usually needs experts who carefully read each step, which is slow and expensive. The authors tested if large language models (LLMs) can help grade these proofs automatically. They made a new dataset of contest solutions graded by experts and tried many existing LLM graders. The results showed current models still make many mistakes. To improve this, the authors created a new system called MABGrader that uses advanced techniques to better match expert grading.
What this means in practice
- •For automated grading teams: Improve automatic grading systems for complex math contest proofs with more accurate and scalable AI verifiers.
- •For math competition organizers: Reduce expert workload by partially automating the review and scoring of candidate contest solutions with AI-assisted verification.
Authors
Azim Ospanov, Zijin Feng, Ding Ding, Chengwu Liu, Haoli Bai, Jiacheng Sun, Lifeng Shang, Farzan Farnia
Abstract
Large Language Models (LLMs) have made remarkable progress on solving competition-level mathematics, yet their ability to verify natural-language mathematical proofs remains relatively underexplored. This gap is especially pronounced for Olympiad-level proofs, which require meticulous assessment of each claim in a long reasoning chain. Current proof verification thus primarily relies on expert inspection, which is costly to scale. In this paper, we present a systematic study of LLM-based verification in this setting. Specifically, we introduce a human-curated dataset of USAMO 2025 candidate solutions with expert grading, and conduct a broad evaluation of existing LLM verifiers across frontier models. We also propose TROJAN, an iterative self-critique pipeline for generating high-fidelity adversarial proofs at scale. Our comprehensive evaluation covers 20 models spanning seven metrics, two prompt styles, three inference methods, and three prompt templates, showing that state-of-the-art models still leave considerable room for improvement. To narrow this gap, we propose MABGrader, a Multi-Armed Bandit framework that reframes grading as arm selection over the discrete score space. MABGrader outperforms the SOTA ProofGrader by 26.4% in Quadratic Weighted Kappa and reduces grading error by 14%. Code and dataset are available at https://github.com/aziksh-ospanov/TROJAN.