Outcome indexed attention vectors stabilize multi outcome learning models
Why shared attention vectors fail: a case for outcome-indexed tuning
Machine LearningNeural and Evolutionary Computing
Summary
Sometimes computer programs learn to pay attention to important signals by using a shared list of weights. But when the program has to predict many different things at once, that shared list breaks down and stops working properly. The authors show this happens because the shared attention becomes unstable during training. They fix the problem by creating separate attention weights for each thing the program predicts, which helps the learning process and makes the program better at understanding patterns. Their new method works well in tests where the old method failed.
What this means in practice
- •For machine learning engineers: Improve multi-label predictive models by using separate attention weights per outcome for better training stability and accuracy.
- •For natural language processing teams: Enhance performance of models predicting multiple linguistic features by avoiding unstable shared attention vectors through outcome-indexed tuning.
Authors
Lenard Dome
Abstract
Dimensional attention in learning is often implemented as a globally shared attention vector, where each stimulus dimension corresponds to a single scalar. These scalars are learned by models through gradient-descent on error, where predictive features acquire more salience. We show that under multi-outcome learning, where models predict more than one outcome, this shared vector becomes unstable; it collapses to its bounds and prevents the models from learning meaningful attentional tunings for learning and generalization. We address this by introducing an outcome-indexed attentional matrix that converts globally shared attentional tuning into an outcome-indexed representation. We present an analysis of the unstable shared vectors and derive the conditions under which it holds. Empirically, three synthetic experiments benchmark the proposed attention matrices and show that they converge to meaningful representations, something shared attention vectors fail to do. These results suggest that outcome-indexed attentional matrices are a general fix for gradient-based attentional processes, which improves models of learning under multi-outcome conditions.