AI helps check complex math proofs with fewer errors
Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs
Artificial IntelligenceComputation and LanguageSymbolic Computation
Summary
Verifying complicated math proofs is hard and can have mistakes. The authors studied a challenge where AI tried to solve advanced math problems and found new correct answers. They created a special system that uses AI and math software together to double-check proofs, making fewer mistakes than usual. This makes it easier to trust and measure progress in AI solving tough math problems.
What this means in practice
- •For software developers: Automate checking complex mathematical code to improve reliability of scientific software.
- •For math software engineers: Integrate AI-assisted proof evaluation pipelines to reduce errors when validating new symbolic math algorithms.
Authors
Alan Sorani, Ashvni Narayanan, Michael Shalyt, Tali Monderer, Rotem Kalisch, Hila Barkan, Elyasheev Leibtag, Shachar Weinbaum, Carsten Schneider, John Campbell, Ido Kaminer
Abstract
As AI's mathematical capabilities rapidly improve, meaningful benchmarks are shifting from university-level problems and math olympiads to research-level mathematics. The Ramanujan Challenge is a benchmark comprised of research-level problems in number theory, including unpublished results and open conjectures. It presented concrete formulas for mathematical constants whose correctness can be checked numerically, while their proofs may require non-obvious mathematics. Here, we analyze the Challenge submissions, measuring the relative success of different AI models, Computer Algebra Systems (CAS), and formal tools. We find correct submissions that are conceptually distinct from the official solutions, and a proof of one of the published open conjectures. Evaluating the submissions exposed a second challenge: judging complex proofs reliably at scale. We introduce an AI-assisted evaluation pipeline for natural language proofs, CAS code, and Lean formalizations that assesses semantic and logical correctness. Compared with the conventional one-shot LLM judgment, our pipeline reduces false-positive verdicts on natural language proofs from 23% to 6% and improves judgment accuracy on supporting code from 46% to 58%. This approach provides a methodology for evaluating research-level mathematics at scale, with potential applications across mathematics and AI for science.