Redpine Science improves scientific search and answer quality significantly
Leveraging a four-quadrant approach for evaluating Redpine Science
Computation and Language
Summary
Finding accurate scientific information is hard and current tools sometimes miss the mark. The authors study Redpine Science, a tool that searches scientific papers and helps AI models answer questions better. Their tests show it finds more relevant papers and helps models answer correctly more often compared to standard web searches or existing tools. They also checked these results carefully with experts to avoid errors from memorized data.
What this means in practice
- •For ai developers: Integrate Redpine Science’s API to improve AI models' access to accurate scientific literature and enhance answer quality on technical queries.
- •For medical data teams: Use Redpine Science to retrieve highly relevant biomedical papers with better precision, reducing noise compared to traditional tools like PubMed.
Authors
Filip Dorm, Leonora Vesterbacka
Abstract
Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context Protocol (MCP) and an API. This report evaluates Redpine Science on two levels: the relevance of the retrieved chunks, and a model's answer when it has access to Redpine Science compared to web search. Both public and expert-validated benchmarks are used. Public benchmarks are a widely accepted way to test model development and are comparable across labs, but risk saturation and memorization. To address this, we complement them with an expert-validated question set. In total, this report presents four evaluations. On ScholarQABench SciFact, the public answer-quality benchmark reported here, an agent with Redpine Science answers 94.4% of claims correctly against 87.6% with no retrieval. On the expert-validated question set, an agent with Redpine Science states 80.1% of the required claims against 70.2% for an agent restricted to web search. On the 668 queries of a public retrieval benchmark whose gold paper Redpine holds, stripped of any model reasoning, Redpine Science places the correct source paper in its top ten results for 83.1% of queries (Recall@10), against 79.3% for the benchmark's creator. A blinded expert relevance panel places Redpine Science's Precision@5 at 75.2% against 39.8% for the PubMed search tool. We release the expert-validated question set and instructions to reproduce every headline result above, at https://github.com/redpine-ai/benchmarks.