All-transistor circuits trained to perform simple in-context learning tasks
An all-transistor physical-learning transformer can be trained to perform in-context learning
Machine Learning
Summary
Learning models like transformers excel at tasks where they can adapt based on context, but typical physical learning systems struggle with these tasks because of circuit limitations. The authors introduce a new transistor-based circuit design that restores a critical two-way flow of information needed for attention mechanisms. Using principles from transistor physics, they simulate this circuit and show it can learn to perform in-context learning tasks with appropriate training methods. This points toward hardware that can physically implement learning processes usually done by software.
What this means in practice
- •For hardware engineers: Develop neuromorphic circuits that physically implement transformer-style learning using all-transistor components for efficient low-power devices.
- •For embedded systems developers: Incorporate transistor-based physical learning modules into embedded hardware to enable adaptive behavior without relying on full digital computation.
Tested on simulated data.
Authors
Michael C. Abbott, Benjamin B. Machta
Abstract
Most systems trained by equilibrium propagation or coupled learning are in the same class as multi-layer perceptrons: while they may be universal function approximators, they struggle to learn associative tasks which are easy for transformers. Equilibrium propagation relies on a bidirectional flow of information through the physical system between input and output. The reverse flow is broken by more complex nonlinear systems, such as those performing multiplication which is needed to express dot-product attention. Here we explicitly restore this bidirectional flow by introducing extra circuit elements for this purpose into a double or dyadic circuit. We then use these ideas to construct an attention mechanism exploiting the exponential current law of sub-threshold MOSFET devices. We show that in simulations this circuit can be trained to perform a simple in-context learning task, despite using realistic reverse components which transmit only an approximate gradient. For this training to succeed we need cross-entropy loss (not mean squared error). We generate nudges appropriate for this loss using a novel floating circuit element, based on the same MOSFET sub-circuit used for softmax. We train the resistors in this circuit using the Lion optimiser, which can be implemented using one capacitor per parameter to store local memory of past gradient estimates.