TPR-Attention for Combinatorial Generalization
2026-08-31 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors studied how deep learning models struggle to easily combine known pieces in new ways, a skill humans do well. They introduced a new method that uses a special attention system working on structured data representations called tensor-product representations (TPRs). Their experiments showed this method helps models better handle new combinations than existing approaches. This suggests that adding explicit structure to attention mechanisms can improve how models generalize to new situations.
systematic generalizationcombinatorial generalizationdeep learningattention mechanismtensor-product representationsinductive biascompositionalityneural architectures
Authors
Melisa Civelekoğlu, Isabeau Prémont-Schwarz
Abstract
Systematic generalization remains a significant challenge in deep learning. In particular, combinatorial generalization - generalizing to new configurations of known factors of variation - is effortless for humans but difficult for standard neural architectures that rely on statistical correlations rather than explicit structural representations. We introduce a new architectural component that embeds structured inductive bias into deep learning: an attention mechanism operating over tensor-product representations (TPRs). Through controlled experiments on compositional tasks, we show that this TPR-attention mechanism outperforms existing architectural components in combinatorial generalization. These results highlight the value of integrating explicit compositional structure into neural attention and point toward a promising path for models capable of systematic generalization.