Hierarchical AI agents mimic fruit fly spontaneous exploration behavior
Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila
Artificial IntelligenceMachine LearningRobotics
Summary
Animals like fruit flies explore their surroundings on their own without clear external goals, but how they do this is not well understood. The authors studied fruit flies’ brain activity and movements, then created computer agents that learn to explore in a similar way without rewards. They found that agents using a method called DIAYN best matched the flies’ behavior and brain signals by learning to select short movement actions hierarchically. This work helps explain how intrinsic motivation drives autonomous exploration in animals and how artificial agents can imitate it.
What this means in practice
- •For robotics engineers: Develop robots that autonomously explore environments by combining hierarchical motor actions guided by intrinsic goals without external rewards.
- •For computational neuroscientists: Use hierarchical intrinsic motivation models to interpret and predict neural activity related to voluntary behavior in animals.
Authors
Declan Kutscher, Reece Keller, Alyn Kirsch, Felix Pei, Aran Nayebi
Abstract
Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.