Rethinking Neural Nonlinearity as Gating
2026-07-03 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors show that many common activation functions in neural networks, like ReLU or Sigmoid, can be seen as special cases of a simpler method called Threshold Gating (TG). They prove TG can replace these activations without losing performance and even help in training smaller or better models. They also introduce a theorem about how many branches TG needs to make deep networks trainable. Lastly, TG could make certain hardware implementations for neural networks more efficient by reducing power and space needs.
activation functionsThreshold Gatingneural networksReLUSigmoidmodel compressionuniversal approximationCNNtransformersanalog in-memory computing
Authors
Muhammad Sabih, Frank Hannig, Jürgen Teich
Abstract
Activation functions are considered an essential primitive for neural nonlinearity, i.e., they enable neural networks to serve as universal approximators. In this paper, we show that this nonlinearity can also be achieved by input-conditioned threshold gating through branches as a universal primitive. We demonstrate that standard activations -- whether piecewise-linear (ReLU, PReLU, Hardtanh) or smooth (SiLU, Sigmoid, Tanh, GELU) -- are in fact instances of a single Threshold Gating (TG) primitive. For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form. We then validate these equivalences by converting pretrained networks across CNNs, transformer-based models, and recurrent architectures, preserving model performance without requiring retraining. Threshold Gating also enables training from scratch that goes beyond replacing existing activations, enabling gains in model compression, performance, and shorter training. We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks. In terms of hardware implementation, TG maps to a unified implementation in the case of analog in-memory systems, addressing the bottleneck of analog-to-digital and digital-to-analog converters (ADC/DAC) that is known to significantly impact power consumption and on-chip area.