Spiking transformer attention works like physical two-level systems
Softmax-Free Attention in Spiking Transformers as the Zero-Temperature Limit of Two-Level Systems
Machine Learning
Summary
Spiking Transformers save energy by skipping softmax normalization used in regular attention models, but this breaks existing ways to understand how attention works using thermodynamics. The authors show that softmax-free spiking attention behaves like a physical system cooled to absolute zero, which explains why optimization can be hard and why energy use hits a fundamental limit. They test their theory and place both standard and spiking attention on the same thermodynamic scale to unify understanding.
What this means in practice
- •For neuromorphic hardware designers: Analyze and predict energy consumption limits and optimization challenges in spiking attention modules using thermodynamic principles.
- •For machine learning engineers: Understand the fundamental difficulties in training softmax-free spiking transformers to improve model design and optimization techniques.
A theory result. No direct application yet.
Authors
Roberto C. Sotero, Pouya Taghipour, Jose M. Sanchez-Bornot
Abstract
Spiking Transformers achieve energy efficiency by discarding softmax normalization in favor of thresholded, binary attention. However, this removal breaks the thermodynamic framework used to analyze attention networks, as the normalized measure and partition function cease to exist. We restore this theoretical foundation by proving that softmax-free spiking attention operates exactly as the zero-temperature limit of a product of independent two-level systems. This correspondence places spiking attention strictly in the frozen ordered phase of the thermodynamic phase diagram for all inputs. Consequently, the entropic cost of organizing attention collapses, replaced by a Landauer-type physical floor governed by action-potential generation. Furthermore, this ordered-phase pinning mathematically explains the optimization difficulty caused by vanishing surrogate gradients. We validate this theory with forward-relaxation experiments, placing both standard and neuromorphic attention mechanisms on a unified thermodynamic axis.