AI agents struggle to fully solve complex physics research tasks
From Research Goals to Scientific Closure: Evaluating Agents Through Evidence-Supported Physics Research
Artificial Intelligence
Summary
This paper looks at whether AI programs can organize and complete scientific research tasks that have clear goals, focusing on challenging physics problems. The authors created a test with 50 physics research tasks, each with specific goals and rules, to see how well AI can generate and check evidence to answer these questions. They found that while AI models can make progress on parts of the tasks, they often fail to completely close the scientific questions because of misunderstandings or missing important conditions. This shows that AI still has trouble finishing complex scientific investigations on its own.
What this means in practice
- •For ai development teams: Improve AI models designed to autonomously conduct and validate complex scientific investigations by using benchmarked task structures and scoring criteria.
- •For physics simulation engineers: Use the benchmark tasks to test and enhance AI-driven frameworks for organizing physics research workflows under scientific constraints.
Authors
Yijia Mao, Song Dai, Tao Chen, Xiangyu Lu, Linqian Zeng, Chenxi Hu, Sitong Zhou, Dongyang He, Xiaodan Mao, Linxuan Zhang, Tong Zhu, Yafu Li, Fangchen Yu, Peng Ye, Feng He, Yu Cheng, Yun Luo
Abstract
Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a benchmark of 50 expert-verified, goal-specified research tasks spanning strong-field, attosecond, and neighboring physics areas. Each task specifies the research objective and essential scientific constraints, while leaving the agent to organize the investigation and determine how to generate and validate the evidence needed to answer the question. A 20-criterion Atomic Score measures local scientific accomplishment, while Research Closure (RC0--RC4) assesses how fully the investigation resolves the specified research question based on whether its indispensable evidence requirements and applicable scope conditions are satisfied. We evaluate seven frontier LLMs, where Claude-Opus-5 achieves the highest mean Atomic Score (6.29/10) but reaches RC4 in only 32.7% of attempts. Across the benchmark, 36.2% of attempts scoring at least 8/10 still fall short of RC4, while all seven models reach RC4 on substantially fewer tasks in all three runs than in at least one. Case-level analysis further illustrates how misinterpretation of generated results and unresolved physical applicability limits can lead to incomplete closure. Together, these results reveal a persistent gap between AI agents' local scientific accomplishments and their ability to achieve complete research closure.