L1 Augmented Attention as an Improved Vector Similarity Metric
2026-07-20 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors found that the usual way transformers measure similarity between words, called dot product attention, mixes two different ideas: direction and size of word vectors, which can be limiting. They created a new method called L1 augmented attention that also considers how far apart the word vectors are by subtracting a learned L1 distance. This helps the model better understand and compare words in a more balanced way. When tested, their method improved performance and was efficient to compute, showing that combining these different geometric measures helps transformer models work better.
TransformerScaled dot product attentionL1 distanceSimilarity metricQueryKeyPerplexityLow dimensional projectionParallel computationHead specialization
Authors
Kurt Godden
Abstract
Scaled dot product attention conflates directional alignment and vector magnitude, limiting its effectiveness as a similarity metric in Transformer models. We introduce L1 augmented attention, a simple and computationally parallelizable modification that subtracts a learned, head specific L1 distance between queries and keys from the dot product score. This hybrid similarity captures complementary geometric information. Dot product rewards directional alignment, while L1 penalizes coordinate deviations. To reduce the cost of L1 computation, we project queries and keys into low dimensional subspaces whose parameters specialize to preserve informative L1 structure. Evaluated on WikiText 2 using a compact transformer, L1 augmented attention achieves up to a 14.5% reduction in perplexity over the original transformer baseline and outperforms an RBF L2 kernel. Analysis of norm variance and learned L1 weights reveals distinct geometric roles across layers and strong head level specialization. These results demonstrate that enriching attention with L1 geometry provides a principled and effective improvement to similarity computation in modern language models, with practical benefits for both accuracy and parallel efficiency.