SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
2026-06-29 • Information Retrieval
Information RetrievalArtificial IntelligenceComputation and LanguageMachine Learning
AI summaryⓘ
The authors created SABER-Math, a new test to see how well computers find math-related information without needing math experts to check. They used many high school math problems and asked AI to summarize solutions and topics, then found relevant documents and rated them based on relevance. They tested different search methods and found that newer AI models work better but still have trouble with complex math like Algebra and Calculus. They also showed that general search tests don't predict how well models handle math searches, so specialized math tests are important.
Information RetrievalMathematical IRLarge Language Models (LLMs)Embedding ModelsRerankingOntologyAlgebraCalculusBenchmarkingSymbol-heavy domains
Authors
Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova, Ivo Petrov, Dimitar I. Dimitrov, Martin Vechev
Abstract
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to directly isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing relevant documents. We address this gap by introducing SABER-Math, the first fully automated benchmark for evaluating mathematical IR without expert annotation. Starting from 283K high-school-level math problems with solutions, SABER-Math builds challenging reranking tasks in three steps: (i) first, LLMs extract concise solution summaries and mathematical topics for each problem; (ii) then, per-query relevant documents are discovered using ontology topic-based and lexical solutions-summary-based similarities, and (iii) finally, a Swiss-style LLM preference tournament produces fine-grained relevance ratings for the documents. We evaluate lexical retrievers, specialized mathematical retrieval systems, and recent embedding models. We find that while modern embedding models substantially outperform classical and math-specific baselines, even the strongest systems struggle in symbol-heavy domains like Algebra and Calculus. Importantly, we show that general-purpose IR benchmarks such as MTEB do not reliably predict mathematical performance, especially for recent embedding models, highlighting the need for math-specific retrieval benchmarks.