Contextual embeddings reveal how meaning shifts in sentences over time
Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction
Computation and LanguageArtificial Intelligence
Summary
Understanding how words change meaning as a sentence unfolds is hard, especially in tricky sentences that mislead readers. The authors show that by looking at how computer-generated word meanings update word by word in a sentence, they can track these changes over time. This approach captures when and where sentences get confusing and can tell apart tricky sentences from clear ones. It suggests that meaning builds up gradually across the whole sentence, not just at the end. These insights could help improve how language models understand complex sentence meanings.
What this means in practice
- •For natural language developers: Detect sentence ambiguity by analyzing how token representations shift incrementally within text inputs during processing.
- •For computational linguists: Use token-wise embedding trajectories to study real-time meaning construction and ambiguity resolution in language data.
Authors
Grayson Wycliffe Storer, Julia Witte Zimmerman
Abstract
Transformer-based large language models (LLMs) such as RoBERTa represent text using contextual word embeddings (CWEs), which alter the embeddings associated with each token based on surrounding context. We construct token-wise incremental trajectories by repeatedly recomputing a token's CWE as successive words are added to a sentence, yielding a representation of how contextualized embeddings evolve as the utterance unfolds. We evaluate this approach using garden-path sentences as a test case with characteristic features. Token-wise trajectories reproduce known features of garden-path processing, including disruption around the critical region, and reliably distinguish garden-path sentences from matched disambiguated controls. We introduce several metrics for quantifying representational displacement across contextual increments and show that trajectory information can be highly predictive of sentence type. We find that ambiguity-related information is recoverable not only from the sentence-level CLS representation but also from ordinary vocabulary tokens, suggesting that utterance-level information is distributed across multiple representational scales. In exploratory analyses, we find qualitatively similar trajectory structures in other ambiguity- and misdirection-related linguistic phenomena. Together, these results establish token-wise incremental trajectories as a promising framework for studying utterance-specific meaning construction using LLMs.