Neural networks balance rules and exceptions when learning patterns
The Dynamics of Quasiregular Neural Learning
Machine Learning
Summary
Learning problems often have main rules with special exceptions that don’t follow those rules. The authors studied how neural networks handle such situations using simple tasks where they know the regular patterns and the exceptions. They found that networks can learn some exceptions early, then forget them briefly when focusing back on the main rules, and later remember them again. This back-and-forth behavior depends on how common the exceptions are and the type of rule, showing a competition between learning regular patterns and exceptions.
What this means in practice
- •For language technology developers: Improve speech or text models by modeling how exceptions compete with regular grammar rules during training.
- •For educational software engineers: Design adaptive learning tools that better reflect how learners temporarily overgeneralize rules before mastering exceptions.
Authors
Matthia Sabatelli
Abstract
Many learning problems combine a dominant regularity with systematic exceptions. Motivated by U-shaped learning in language acquisition, we study this interaction in controlled quasiregular regression problems where regular and exceptional solutions are explicitly known. Neural networks can partially acquire exceptions, subsequently regress toward the dominant regularity, and finally recover. This overregularization becomes substantially stronger when exceptions are rare, despite their early acquisition, but does not emerge equally across all regularities considered. Our results isolate a simple form of competition between regularities and exceptions during neural learning.