Language models work well without long context windows
You don't need long context
Artificial Intelligence
Summary
Generating later parts of a text often takes more computing time than earlier parts. The authors show that large language models can perform almost as well on many tasks even when only focusing on short pieces of context frequently. They demonstrate this especially in tasks like software engineering, where the model doesn't need to consider a long history to do well. This suggests that using very long contexts might not be necessary to get good results.
What this means in practice
- •For software developers: Build coding assistants that keep only short context windows without losing accuracy, speeding up response time and saving resources.
- •For machine learning engineers: Design language model deployments that reduce computational costs by maintaining short context while preserving performance on long-input tasks.
Authors
Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov
Abstract
Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.