Modeling timing of new and surprising words in language generation
Modeling the Arrival Time of Information-Bearing Events in Natural Language
Computation and Language
Summary
This work looks at when important pieces of information appear in conversations or texts, like new words or unexpected words. The authors use a method called neural temporal point processes combined with language model context to predict when these events happen. They find that AI models create new words more frequently than humans, but both produce surprising words at similar rates. Their findings give new insights into how language unfolds over time and highlight differences between human and machine language generation.
What this means in practice
- •For chatbot developers: Improve chatbot responses by predicting timing of new and unexpected words to generate more human-like conversation flow.
- •For speech recognition teams: Enhance speech-to-text systems to better anticipate when new or surprising words are likely to occur, helping with transcription accuracy.
Authors
Leyi Lao, Yulin Ou, Yang Xu, Hao An
Abstract
Understanding how information is distributed over time is fundamental to modeling natural language. While existing empirical studies primarily rely on post-hoc analysis of information-theoretic metrics, such as surprisal, little is known about the generation processes under the surface. We propose a novel approach for addressing this gap using the tool of neural temporal point processes (TPPs) by modeling the key information-bearing events in text sequences with learnable event intensity functions. Our model takes the history of event intervals and the contextual semantic representations from LLMs as input, and predicts the future arrival times for two types of events: new-word and high-surprisal-token. The experimental results reveal an interesting distinction in event intensities: LLMs tend to produce new words at significantly higher rates than humans, while high-surprisal tokens do not exhibit the difference. A decaying pattern of intensity over time is found in new-word events for both humans and LLMs, but not in high-surprisal-token events. Our findings revisit and extend previous studies that use simpler methods; provide supplementary evidence for theories about constrained context representation inherent in human language production; and find a new angle to view human and LLM generation differences.