Synthetic world reveals how agents learn and diagnose systems differently
LUCA: A Synthetic World for Experimental Reasoning
Artificial Intelligence
Summary
Sometimes computer programs can fix problems but for the wrong reasons, which makes it hard to trust their decisions. The authors created LUCA, a made-up world where programs can run experiments, learn rules, find hidden problems, and try to fix them. This setup helps check not just if the program gets the right answer, but how it got there. They tested several big language models and found that even if the results look similar, the ways they choose experiments and diagnose problems can be very different.
What this means in practice
- •For ai system developers: Test and improve how AI systems learn from experiments and diagnose complex problems using a controlled synthetic environment.
- •For software testing teams: Use LUCA to evaluate fault diagnosis and system rescue strategies to better understand experimental reasoning in automated testing.
Authors
Anna Gogleva, Thomas Charman, Martin Balla, Karl Gemayel
Abstract
An experimental agent can succeed for the wrong reasons. Evaluations, however, often focus on the outcome, not the experiments, evidence, and inferences that produced it. We introduce LUCA, a configurable synthetic world for evaluating this process under fully known ground truth. Agents learn unknown regulatory systems through budgeted experimentation, then predict unseen interventions, diagnose hidden faults, and attempt rescues. This setup allows us to evaluate experiment selection, prediction, diagnosis, and intervention separately. Across seven frontier LLMs and three baselines, these capabilities dissociate sharply. Agents frequently rescue malfunctioning systems while diagnosing the wrong fault; diagnosis varies 7x across models while prediction varies only 1.4x. Models also adopt surprisingly different experimental strategies, and greater exploration does not reliably yield better performance. In one failure mode, agents often perform the experiment needed to distinguish competing causes, but still fail to use the resulting evidence. LUCA turns these hidden failures into measurable behaviours, providing a controlled environment to study where experimental agents succeed and fail. Code: https://github.com/RelationRx/luca.