Long-context language models struggle with tracking and pattern learning
LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking
Computation and Language
Summary
Language models that handle very long inputs are good at remembering facts but have trouble with more complex tasks like tracking changes over time or recognizing patterns from many examples. The authors created new tests called LEAPS to better measure these struggles by using very long texts and tasks beyond just recalling facts. They found many models get worse as inputs get longer, especially on tasks needing more thinking, and sometimes versions designed to think harder don’t perform well. This work helps highlight where future models need improvement for tasks requiring understanding and adapting over long text stretches.
What this means in practice
- •For software developers: Build language model features that better track state changes and learn patterns over long project histories for coding and automation tasks.
- •For market analysts: Use language models evaluated on LEAPS to improve forecasting of market trends by capturing long-term dependencies in data.
Authors
Howard Yen, Nitya Mani, Alex Renda, Christopher S. Hardin
Abstract
Long-context language models (LCLMs) power agents that act over long trajectories, from improving a codebase to running research experiments to forecasters that condition on lengthy histories. These applications draw on three distinct long-context capabilities: recall, where the answer is a fact present in the context; state tracking, where the outcome can be computed by performing a series of transformations to a specified state; and pattern learning, where the goal is to make a prediction by sampling from a distribution inferred from many observations. Existing benchmarks primarily focus on the first capability. In this work, we introduce LEAPS (Long-context Evaluation of Adaptive Pattern learning and State tracking), a collection of 12 novel datasets, spanning agentic workflows, behavioral modeling, and forecasting of market and world events. By construction, LEAPS is resistant to data contamination, extends to millions of input tokens, and uses reliable metrics. We evaluate 30 API and open models from 8K to 512K input tokens. LEAPS distinguishes models that perform near perfectly on recall tasks, finding that such models' state tracking and pattern learning abilities can vary significantly and degrade much more dramatically with increasing input lengths--even for pattern learning, where more observations should make learning a theoretically easier task. Moreover, thinking models often underperform their base counterparts, exhausting their generation budget or oversimplifying the task. We hope LEAPS can guide the development of the next generation of LCLMs.