Cognitive algorithms improved with human and AI guided program fixes
Hypothesis-guided discovery of cognitive algorithms via program refinement
Artificial Intelligence
Summary
Understanding how people think and solve problems is hard because human thinking can be complicated. This paper shows a way to improve models that explain human thinking by combining expert-made programs with large AI language models that suggest and check small program changes. These improvements better match how people actually behave in problem-solving tasks. The method helps find key changes that make models more accurate and flexible.
What this means in practice
- •For cognitive scientists: Improve computational models of human problem-solving by integrating expert knowledge with AI-driven program updates to better explain experimental behavior.
- •For ai algorithm developers: Use hybrid program refinement to create more accurate algorithms that mimic human reasoning patterns in cognitive tasks.
Authors
Huiwen Alex Yang, Mark K. Ho, Bill D. Thompson
Abstract
Developing cognitive models of algorithmic reasoning from behavioral data is a central problem in cognitive science that challenges current methods. Traditional approaches to cognitive modeling are interpretable and benefit from human expertise, but lack flexibility and scalability. Emerging techniques using large language models (LLMs) for de novo generation of cognitive models are scalable and flexible, but lack a role for human expertise and have mostly been applied to simpler tasks than algorithm recovery. We propose a hybrid system that treats discovery of cognitive algorithms as a program refinement problem. Human-created cognitive models are expressed as probabilistic programs and provided to a system of LLM agents with a mandate to: identify mismatches between model and behavior; propose code-level modifications within researcher-specified constraints; and verify structural fidelity. Revisions propagate to a probabilistic inference module that performs inference for latent variables and data likelihood computations. We evaluate the pipeline on human behavior in a problem-solving paradigm that exposes a variety of cognitive algorithms. Revised models consistently improve model fit relative to ancestral models and reveal a small set of recurring innovations that capture meaningful behavioral variability in this task.