Spiking networks improve learning with sparse rewards in robots
SpikeCredit: Temporal Credit Carrier for Reinforcement Learning with Sparse Rewards
Machine Learning
Summary
Reinforcement learning often struggles when rewards are rare, making it hard to know which actions lead to success. The authors show that spiking neural networks, which mimic brain activity patterns, can better track and assign credit to important past actions through special signals they call Temporal Credit Carriers. They introduce SpikeCredit, a system that uses these signals to improve learning in simulated robot tasks with sparse rewards, achieving much better performance than other spiking-based methods. Their results suggest that brain-inspired timing signals can help machines learn more effectively when feedback is infrequent.
What this means in practice
- •For robotics engineers: Improve robot control learning when rewards or feedback signals are rare or delayed by using spike-based credit assignment methods.
- •For embedded systems developers: Design energy-efficient neuromorphic processors that handle sparse feedback signals for decision-making tasks more effectively.
Authors
Yingchao Yu, Pengfei Sun, Wenxuan Pan, Wei Chen, Yitian Hong, Kuangrong Hao, Yaochu Jin
Abstract
Reinforcement learning (RL) with sparse rewards is challenging because delayed outcomes provide little guidance about which intermediate computations caused success or failure. We argue that reliable credit assignment requires policy dynamics that preserve and expose credit-relevant information over time, a role we formalize as Temporal Credit Carriers (TCCs) and that spiking neural networks (SNNs) naturally fulfill through graded membrane traces and event-driven spikes. Based on this hypothesis, we propose SpikeCredit, an SNN-based framework for RL with sparse rewards that first performs task-adaptive TCC selection and then closes the loop between a fast TCC-reading pathway, where self-motion feedback constraint uses local behavior-grounded cues to constrain transition-level credit recovery, and a slow TCC-writing pathway, where credit-targeted trace alignment feeds recovered credit back into the actor to make future TCC dynamics more credit-readable. Across sparse-reward MuJoCo tasks, SpikeCredit improves Last10 return over sparse SNN baselines by +1169% on Ant, +953% on Hopper, +723% on Swimmer, and +1781% on Walker2d, and exceeds the dense-reward baseline on Swimmer by +113%. Mechanistic analyses further show substantially stronger alignment with dense rewards than the sparse SNN baseline. These results position spiking dynamics as credit-preserving substrates for sparse-reward RL.