Neural networks learn addition patterns using Fourier frequency methods
Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks
Artificial Intelligence
Summary
Training neural networks to do addition with modular arithmetic often leads them to use special patterns based on frequencies, like musical notes. The authors explain how the training process naturally picks these patterns by looking at the probability structure of the training data. They also show why noisy or corrupted training examples can sometimes make the network learn faster early on, even if those examples don't lead to correct answers in the end. Their approach also works for other tasks like XOR, another common logic problem.
What this means in practice
- •For machine learning engineers: Improve neural network training on modular arithmetic tasks by analyzing data statistics to predict network behavior and accelerate convergence.
- •For signal processing developers: Design frequency-based algorithms and models in neural networks by using probability signature methods demonstrated on addition and XOR tasks.
Authors
Yunji Wang, Junjie Yao, Linyu Liu, Pinyan Lu, Zhi-Qin John Xu
Abstract
Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.