Clockless chip powers large-scale parallel reinforcement learning decisions
Massively Parallel Reinforcement Learning with a Chaotic Reconfigurable Clockless Chip
Neural and Evolutionary Computing
Summary
Reinforcement learning can use a lot of energy and get slower when scaled up, especially when relying on hardware devices that depend on random signals. The authors built a special chip without a clock that uses many tiny units producing independent random bits simultaneously, creating complex and unpredictable behavior. This chip can handle thousands of decision-making tasks in parallel more efficiently than past devices. Their approach could lead to faster and more energy-efficient systems for decision-making tasks.
What this means in practice
- •For embedded systems engineers: Develop energy-efficient hardware modules that perform large-scale parallel decision-making tasks using distributed Boolean chaos on reconfigurable chips.
- •For high-frequency trading teams: Use high-throughput chaotic entropy sources to accelerate rapid decision-making algorithms for large-scale portfolio optimization.
Authors
Eric Oliveira-Gomes, Damien Rontani
Abstract
Hardware accelerators based on physical dynamical systems offer an attractive route toward energy-efficient reinforcement learning applications. However, their scalability is challenging because it requires many statistically independent entropy sources. Here, we introduce a quasi-analog decision-making architecture based on asynchronous Boolean networks (or lattices) implemented on a clockless reconfigurable chip. Each node in the network consists of a single logic element that acts as an autonomous entropy source. This architecture gives rise to distributed Boolean chaos, in which a spatially coupled network generates parallel streams of chaotic Boolean transitions with very low statistical dependence between nodes. We experimentally demonstrate parallel decision-making on a 1024-armed bandit problem, which is beyond the scale of previous hardware implementations, while significantly improving power-law scaling performance. Separately, we scale the proposed entropy source to 5120 parallel channels, yielding an aggregate sample generation rate of 2.14 TS/s. Our solution is implemented on a commercial reconfigurable CMOS chip and offers high integration density and ease of programmability. Our results pave the way for using distributed Boolean chaos as a valuable hardware substrate for large-scale reinforcement learning and for the development of fully integrated, high-throughput decision-making accelerators.