Papers for

math software engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

AI helps check complex math proofs with fewer errors

Ramanujan Challenge Results: AI-Assisted Evaluation Pipeline for Multi-Format Proofs

Abstract: As AI's mathematical capabilities rapidly improve, meaningful benchmarks are shifting from university-level problems and math olympiads to research-level mathematics. The Ramanujan Challenge is a benchmark comprised of research-level problems in number theory, including unpublished results and open conjectures. It presented concrete formulas for mathematical constants whose correctness can be checked numerically, while their proofs may require non-obvious mathematics. Here, we analyze the Challenge submissions, measuring the relative success of different AI models, Computer Algebra Systems (CAS), and formal tools. We find correct submissions that are conceptually distinct from the official solutions, and a proof of one of the published open conjectures. Evaluating the submissions exposed a second challenge: judging complex proofs reliably at scale. We introduce an AI-assisted evaluation pipeline for natural language proofs, CAS code, and Lean formalizations that assesses semantic and logical correctness. Compared with the conventional one-shot LLM judgment, our pipeline reduces false-positive verdicts on natural language proofs from 23% to 6% and improves judgment accuracy on supporting code from 46% to 58%. This approach provides a methodology for evaluating research-level mathematics at scale, with potential applications across mathematics and AI for science.

Thu 29 OctArtificial IntelligenceComputation and LanguageSymbolic Computation
The gist
Verifying complicated math proofs is hard and can have mistakes. The authors studied a challenge where AI tried to solve advanced math problems and found new correct answers. They created a special system that uses AI and math software together to double-check proofs, making fewer mistakes than usual. This makes it easier to trust and measure progress in AI solving tough math problems.
Open → 2610.40551v1

Endless Exam measures AI math progress with scalable benchmarks

The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence

Abstract: We introduce the Endless Exam, a benchmark for measuring mathematical progress from today's models toward artificial superintelligence through fourteen parameterised construction families. Each submitted object is checked automatically for validity and assigned a relative quality score against a published frontier or construction baseline, without capping improvements at $1$. The families draw on open mathematical problems for long-term targets and generate new instances at larger parameters, where compact certificates keep large constructions verifiable. Across eight models evaluated on 69 distinct instances, continuous quality scores distinguish performance even though no evaluated system surpasses a published frontier. Size-quality curves show how construction quality changes as problem size increases. We release the generators, verifiers, references, model responses and analysis to support continued measurement before and beyond human frontiers.

Mon 21 SeptArtificial Intelligence
The gist
Measuring how well AI models can solve complex math problems is hard, especially as problems get bigger. This paper introduces the Endless Exam, a set of math challenges that grow in size and difficulty to track AI progress toward very advanced intelligence. The authors provide tools that check answers automatically and score them relative to the best known solutions, helping compare different AI systems fairly. They tested eight AI models on many problems but found none beat the current best human or machine results. Their system lets others keep testing future AI as it improves beyond what humans can do.
Open → 2609.24555v1