Reducing activation bias slows learning but improves final AI performance
Rethinking Activation Functions: Reducing Bias Can Improve Learning
Machine LearningArtificial Intelligence
Summary
Many popular activation functions in AI models tend to produce mostly positive outputs, causing a bias that affects how the model learns. The authors found that this bias helps models learn quickly at the start but leads to unstable learning and worse final results. When the bias is reduced, learning is slower initially but becomes smoother and ends with better overall performance. Tests on large vision and language AI models confirmed this pattern.
What this means in practice
- •For deep learning engineers: Improve model training by selecting or designing activation functions with reduced bias to enhance late-stage performance.
- •For machine learning platform developers: Integrate bias-reducing activation functions into frameworks to offer more stable and effective training options.
Authors
Zirui Peng, Heyi Zheng, Ziming Liu
Abstract
Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.