Language model harnesses tested for accuracy and speed on long contexts
LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
Computation and Language
Summary
Language models can use special methods called harnesses to handle very long pieces of text better, but current tests don’t show clear differences between these methods. The authors created a new set of challenges where models must search and think strategically over large amounts of information to answer questions accurately and efficiently. They found that even strong combinations of models and methods struggle with these tasks, and how efficiently a model works depends a lot on the specific method used. This shows that testing both accuracy and speed is important when evaluating how well language models handle long texts.
What this means in practice
- •For software engineers: Optimize long-text processing in AI systems by selecting efficient model harnesses tailored to task-specific retrieval and reasoning strategy.
- •For data center operators: Improve compute resource allocation by understanding different harness efficiency when running long-context language model tasks.
Authors
Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen, Xi Ye
Abstract
Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a challenging search problem and different accuracy--cost tradeoffs across processing strategies. For example, one task requires identifying every person satisfying several conditions using evidence scattered across documents; strategically checking the most selective condition first can narrow the search before verifying the remaining conditions. We evaluate multiple families of frontier language models with four state-of-the-art harnesses. Our benchmarks remain challenging even for strong model--harness combinations: the best reaches 68\% macro-average accuracy across four evaluation suites. More importantly, we find that the same underlying model can exhibit markedly different efficiency under different harnesses. Our results establish efficiency as an important axis for long-context evaluation and provide a testbed for developing harnesses that process context strategically rather than exhaustively.