Scientist world creates runnable environments for ai research papers
Scientist World: Grow a Research World for AI-Scientists
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
AI researchers have tools to read scientific papers but can't easily run experiments from them. The authors built Scientist World, where every research paper is turned into an interactive environment an AI can explore and experiment within. This helps AI scientists test their ideas and improve over human results in many cases. The system also records all trials and outcomes, helping train better AI models for scientific discovery.
What this means in practice
- •For ai research platform developers: Build interactive scientific experiment environments automatically from research papers to enhance AI model training and evaluation.
- •For machine learning engineers: Use recorded AI research traces from diverse scientific questions to improve model creativity and problem-solving skills.
Authors
Taiming Lu, Sachin Konan, Zhuang Liu
Abstract
AI-Scientists arrived, but a world for them to grow in did not. Human scientists take theirs for granted $-$ centuries of open problems and shared ideas, waiting in the scientific literature. To AI-Scientists, that world is readable but not runnable, so we build Scientist World for them: every research paper becomes an executable environment. Our autonomous pipeline extracts each paper's open-ended research question and reproduces its proposed method as the baseline; an AI-Scientist entering the environment is evaluated against what the human authors achieved. We release the first 1,000 verified environments. With the world as an evaluation, we found that frontier AI-Scientists can already surpass the human researchers 74% of the time. As a record of how AI does science, it contains 6,400 hours of research traces, with every idea, experiment, failure, and success. As a training ground, we are able to raise (+158%) a small model to match frontier models in research ideation. At last, AI-Scientists' own world has arrived, grown from and growing with science itself.