Mathematical reasoning models distinguish equivalent rule orderings distinctly
Order-Invariant Answers, Order-Sensitive Representations in Mathematical Reasoning
Machine LearningArtificial IntelligenceComputation and LanguageSymbolic Computation
Summary
People expect that changing the order of math rules without changing their meaning should lead to the same answer. This paper finds that artificial intelligence models can give the right answer but still think about the rules in different ways depending on their order. The researchers discovered that better models represent different rule orders more distinctly inside their
What this means in practice
- •For ai system designers: Improve the internal representations of rule-based models to increase accuracy on multi-step math reasoning tasks with reordered inputs.
- •For software developers: Use insights on representation invariance to build more robust mathematical reasoning modules that handle rule order variations effectively.
Authors
Zhixu Silvia Tao
Abstract
Reordering a set of mathematical rules without changing its meaning should preserve the correct answer, but must a model's internal representations stay invariant too? We investigate this question using synthetic multi-step function-composition problems, each presented under multiple rule orderings with the same correct answer. We measure accuracy and permutation signal-to-noise ratio (SNR), which quantifies how distinctly ordering patterns are represented relative to variation across problem instances. Across 16 language models ranging from 1B to 8B parameters, we find a pattern: models that solve reordered problems more accurately represent different rule orderings more distinctly. Layer-averaged permutation SNR is positively rank-correlated with accuracy in every synthetic setting we evaluate, with Spearman correlations reaching 0.86. These findings highlight a distinction between answer invariance and representation invariance: successful mathematical rule composition can accompany distinct internal representations between equivalent rule orderings. This motivates distinguishing answer invariance from representation invariance, and offers a representational perspective on mathematical reasoning beyond answer accuracy alone.