MLPs and transformers show different learning on noisy parity features
Sometimes less is more: Investigating inductive biases of MLPs and Transformers on noisy k-parity features
Machine Learning
Summary
Some computer models called MLPs and Transformers can learn differently when given data with tricky noisy patterns called k-parity features. The authors found Transformers handle noisy data better but struggle with rare complex patterns, while MLPs get better with more data and learn rare patterns more easily. They also saw that giving Transformers more varied training examples sometimes slows their learning and reduces final accuracy. This study helps reveal when and why these models behave differently and how to best train them for different data conditions.
What this means in practice
- •For machine learning engineers: Choose training data size and model type to balance learning speed and accuracy on noisy or rare features.
- •For data platform developers: Design synthetic testbeds with controllable feature properties to evaluate model robustness and training schedules efficiently.
Authors
Alasdair Paren
Abstract
We empirically investigate how MLPs and Transformers differ when trained on data sets containing one or more features encoded by noisy k-parity. Noisy k-parity gives us hard synthetic data sets in which we can control the predictivity and availability of every feature. Across a range of data distributions, data scales and model sizes, we find large differences in the training dynamics and inductive biases of the two architectures: Transformers are more robust to noisy features but struggle to learn complex, low-availability features, whereas MLPs show the opposite trend and improve steadily with more data. We also find that, under a fixed training budget, increasing the number of unique training examples can slow Transformer optimisation enough to reduce final accuracy, so that a smaller data set performs better. Training-accuracy results show this is an optimisation-speed effect; it shrinks, but does not vanish, when the budget is scaled with the data set size. By giving precise control over feature predictivity and availability, this testbed offers a clear starting point for understanding when and why Transformers and MLPs learn differently, and for designing data and training budgets that exploit those differences.