Deep spiking Q-networks improve decision making with less computation
Common-Mode Errors Limit Low-Timestep Deep Spiking Q-Networks
Neural and Evolutionary ComputingMachine Learning
Summary
Spiking neural networks are energy-efficient models that make decisions by simulating brain-like activity over short time steps. However, using very few time steps often leads to poor decision-making. The authors found that this happens because errors common to all choices confuse the learning process. They created a method that uses a simple extra network to correct these common errors during training, which improves performance without extra cost when actually making decisions. Their approach works better than existing methods on popular game tests with very short time steps.
What this means in practice
- •For embedded systems engineers: Create more energy-efficient reinforcement learning controllers on edge devices by reducing the simulation steps needed for spiking neural networks.
- •For video game ai developers: Implement fast and low-energy deep spiking Q-networks for real-time decision making in games requiring rapid updates.
Authors
Zijie Xu, Bingrui Guo, Yiding Sun, Yiting Dong, Zhile Yang, Zhaofei Yu
Abstract
Spiking neural networks (SNNs) offer sparse and event-driven computation, making them attractive for energy-constrained reinforcement learning (RL) on edge devices. In value-based RL, deep spiking Q-networks (DSQNs) combine such efficiency with action-value estimation for decision making. However, existing DSQNs often require multiple simulation timesteps for competitive performance, increasing computational and energy costs, whereas reducing the timesteps can cause substantial performance degradation. We investigate this degradation from the perspective of Q-value estimation errors. By decomposing errors across actions into common-mode and differential-mode components, we find that low-timestep DSQNs suffer disproportionately from common-mode errors shared across action values, which are particularly detrimental to temporal-difference learning through bootstrapped targets. Based on this finding, we propose Common-Mode Compensation Deep Spiking Q-Network (CMC-DSQN), which uses an auxiliary ANN to compensate for common-mode errors in the SNN outputs. At inference, greedy action selection can be performed directly from the SNN outputs, allowing the auxiliary ANN to be completely removed and preserving the energy efficiency of SNNs. Extensive experiments on Atari and MiniAtar environments demonstrate substantial performance improvements under low-timestep settings. CMC-DSQN outperforms state-of-the-art DSQN baselines by nearly $20\%$ at $T=2$ and further surpasses the ANN baseline at $T=4$.