Shared attention vectors cause problems in multi-outcome learning models
Why shared attention vectors fail: a case for outcome-indexed tuning
Machine LearningNeural and Evolutionary Computing
Summary
When computer models learn to predict more than one result at the same time, they usually use shared attention settings to decide what details matter most. However, this shared approach can become unstable and stop the models from learning properly. The authors found that using separate attention settings for each outcome helps the models focus better and learn more meaningful patterns. They tested this idea with experiments and showed it works better than the shared method. This insight can help improve how models learn when dealing with multiple tasks.
attention vectormulti-outcome learninggradient descentattentional tuningpredictive featuresoutcome-indexed matrixlearning generalizationsynthetic experiments
Authors
Lenard Dome
Abstract
Dimensional attention in learning is often implemented as a globally shared attention vector, where each stimulus dimension corresponds to a single scalar. These scalars are learned by models through gradient-descent on error, where predictive features acquire more salience. We show that under multi-outcome learning, where models predict more than one outcome, this shared vector becomes unstable; it collapses to its bounds and prevents the models from learning meaningful attentional tunings for learning and generalization. We address this by introducing an outcome-indexed attentional matrix that converts globally shared attentional tuning into an outcome-indexed representation. We present an analysis of the unstable shared vectors and derive the conditions under which it holds. Empirically, three synthetic experiments benchmark the proposed attention matrices and show that they converge to meaningful representations, something shared attention vectors fail to do. These results suggest that outcome-indexed attentional matrices are a general fix for gradient-based attentional processes, which improves models of learning under multi-outcome conditions.