Simulated cognitive models can help discover human decision theories
Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humans
Artificial Intelligence
Summary
It is often hard to study how people make decisions without testing them directly. The researchers used a computer simulation of human decision behavior called Centaur to generate data. They then ran an automated system, AutoCog, that created and tested different theories purely on this simulated data. Interestingly, the theories found this way worked well when applied to actual human behavior, suggesting simulations can help explore ideas before testing on people. This approach means imperfect simulations can still be useful by focusing on key differences between theories rather than matching every detail of human behavior.
behavioral foundation modelslarge language modelscognitive sciencesimulated datamulti-attribute decision-makingtheory discoveryautomated experimentationmodel generalizationhuman behavior simulationautomated cognitive scientist
Authors
Akshay K. Jagadish, Younes Strittmatter, Nori Jacoby, Eric Schulz, Nathaniel Daw, Thomas L. Griffiths, Suyog H. Chandramouli
Abstract
Behavioral foundation models have been proposed as stand-ins for human participants across settings, but it is unclear whether theories discovered on them generalize to humans or merely characterize the simulator. We ran the Automated Cognitive Scientist (\textsc{AutoCog}), a closed-loop discovery system in which LLM agents design theory-discriminating experiments, collect responses, arbitrate between competing theories, and synthesize successors, entirely on behavior simulated by Centaur, a foundation model of human behavior. In a multi-attribute decision-making setting, the theories \textsc{AutoCog} found on Centaur generalized to human data: they outperformed canonical theories on ten held-out experiments and were rivaled only by theories found by running the same loop on people. We argue that this succeeds despite the simulator's inevitable imperfections because a discovery loop that arbitrates between competing theories demands less of its simulator than estimation does. The simulator only needs to capture the regularities that distinguish the theories, and not necessarily reproduce behavior precisely. Imperfect simulators can therefore widen the search over theories, with human data then testing whether the surfaced theories generalize.