Sensory realism improves cognitive model alignment with human behavior

Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition

Artificial Intelligence

Summary

Many computer models of thinking use simple, symbolic data rather than realistic sensory inputs, assuming this simplification doesn’t matter much. The authors compare models trained with abstract symbols to those using detailed sensory-like inputs on memory and decision tasks. They find models with realistic sensory inputs behave more like people than the simpler ones, although all models still differ from actual human behavior. This suggests that including detailed sensory information helps models better match human responses in tasks involving visual memory.

What this means in practice

  • For cognitive modelers: Build more behaviorally consistent cognitive models by using sensory-realistic inputs rather than abstract symbolic data.
  • For machine learning developers: Improve the design of memory-dependent decision-making AI by incorporating detailed sensory inputs for closer alignment with human patterns.

Authors

Hao Yuan Bai, Xiaoxuan Lei, Zihan Weng, Motahareh Pourrahimi, Lucas Gomez, Pouya Bashivan

Abstract

Computational models of cognition have traditionally relied on abstract symbolic inputs that approximate sensory experiences with simplified vector representations. This simplification reflects historical computational constraints and a long-held assumption that the details of sensory realism are incidental to the study of cognitive processes. Recent advances have enabled the development of sensory-realistic models capable of performing diverse cognitive tasks, yet whether sensory realism meaningfully shapes cognition has not been formally established. We address this gap through a large-scale empirical comparison of abstract and sensory-realistic neural network models trained on a battery of memory-dependent decision-making tasks. Our results show that sensory-realistic models mirror human behavioral response patterns significantly more than their abstract counterparts, though a substantial gap to human internal behavioral consistency remains even for the best tested models. By analyzing the internal dynamics of these systems, we find that the learned recurrent weights are not the main source of this divergence. In fact, frozen abstract-trained weights driven by sensory-realistic inputs recover the human consistency of sensory-realistic models. The resulting dynamics are also within the seed-to-seed variability of sensory-realistic models. Together, our findings argue that across a broad set of visual working-memory tasks, sensory realism accounts for a substantial portion of the model-human behavioral agreement, more than the choice of sequence architecture and comparable to the choice of vision encoder. These results suggest that findings obtained from abstract task models in neuroscience should be more carefully interpreted in light of their potential behavioral limitations.