Agents struggle to form new scientific theories from scattered clues
ULTRADISCOVERY: Abductive Exploration in an Interconnected, Epistemically Open Universe
Machine LearningArtificial Intelligence
Summary
Scientific discovery often starts by piecing together clues from different areas to find a new way to explain something. The authors study how AI agents handle this when they don't fully know the right concepts or when evidence is spread out. They create a test world where agents must update their understanding and predict unseen events across different topics. The study shows that agents find it especially hard to build new ways of describing things that can connect all the scattered evidence and make accurate predictions.
What this means in practice
- •For ai system builders: Create AI that better revises and extends scientific models by integrating diverse evidence across fields.
- •For data integration engineers: Improve tools that combine scattered data sources by understanding challenges in composing distributed evidence.
Authors
Weihan Li, Tianshi Zheng, Yangqiu Song, Ginny Y. Wong, Simon See
Abstract
Scientific discovery often begins when scattered clues call for a new way of describing the world. Such abductive exploration can require constructing the representation in which an explanation is stated, when the world is epistemically open, and composing evidence scattered across contexts, when it is structurally interconnected. Existing benchmarks rarely separate these two demands or control them independently. We introduce ULTRADISCOVERY, an interactive world of five domains in which an agent revises an initially successful theory and predicts the outcome of an unseen cross-domain intervention. A $2 \times 2$ design leaves the representation open or discloses it, and leaves the evidence distributed or aligns it, with the latent dynamics fixed. With the representation open, agents across eleven models often retract the axiom they were taught, and none introduces the unobserved entity or rewrites the variables that a replacement requires. Disclosure triples intervention requests and adds about one of the eighteen findings the world affords, and alignment adds less. Two vendor-harness systems carry discovery into more domains, and one of them rewrites the variables in Open episodes. No system makes the exact prediction within 200 paid actions. At larger budgets one exact prediction appears with both aids, while every Open episode remains inexact. The results locate the difficulty in the step from accumulating evidence to composing it into a representation that transfers.