Enhancing Hyperspace Analogue to Language (HAL) Representations via Attention-Based Pooling for Text Classification
2026-03-20 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors improved a method called HAL, which uses word relationships to understand sentence meaning, by changing how it combines words into sentence summaries. Instead of treating every word equally, they taught the model to pay more attention to important words and less to common, unimportant ones. They first made the data smaller and easier to work with before applying this attention method. Tested on movie reviews, their approach better identified sentiment, making results more accurate and easier to interpret.
Hyperspace Analogue to Language (HAL)word co-occurrence matrixmean poolingattention mechanismTruncated Singular Value Decomposition (SVD)latent spacesentiment analysisstop-wordsIMDB datasetmodel interpretability
Authors
Ali Sakour, Zoalfekar Sakour
Abstract
The Hyperspace Analogue to Language (HAL) model relies on global word co-occurrence matrices to construct distributional semantic representations. While these representations capture lexical relationships effectively, aggregating them into sentence-level embeddings via standard mean pooling often results in information loss. Mean pooling assigns equal weight to all tokens, thereby diluting the impact of contextually salient words with uninformative structural tokens. In this paper, we address this limitation by integrating a learnable, temperature-scaled additive attention mechanism into the HAL representation pipeline. To mitigate the sparsity and high dimensionality of the raw co-occurrence matrices, we apply Truncated Singular Value Decomposition (SVD) to project the vectors into a dense latent space prior to the attention layer. We evaluate the proposed architecture on the IMDB sentiment analysis dataset. Empirical results demonstrate that the attention-based pooling approach achieves a test accuracy of 82.38%, yielding an absolute improvement of 6.74 percentage points over the traditional mean pooling baseline (75.64%). Furthermore, qualitative analysis of the attention weights indicates that the mechanism successfully suppresses stop-words and selectively attends to sentiment-bearing tokens, improving both classification performance and model interpretability.