Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing
2026-08-07 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors found that AI models used for testing scientific hypotheses often make small mistakes that lead to wrong results, even when their calculations are correct. They created a new test called P-Bench with 425 real-world tasks to better check if these AI models understand when a p-value is valid. They also developed a new AI called Fisher-R1, which learns from practice and feedback to improve its hypothesis testing skills. Fisher-R1 did better than existing models, showing that training with clear statistical goals helps make AI more reliable at these tasks. Overall, the authors highlight weaknesses in current AI reasoning for statistics and present a way to fix it.
hypothesis testinglarge language modelsp-valuestatistical inferencereinforcement learningbenchmarkFisher-R1synthetic tasksempirical sciencestatistical reasoning
Authors
Jiacheng Miao, Jin Mu, Guanhua Chen, James Zou
Abstract
Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeekV4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.