Corpus task complexity affects large text model reasoning performance
No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow
Computation and LanguageArtificial Intelligence
Summary
Some questions about a big collection of text are easier to answer than others, depending on how complicated the task is as the text grows. The authors introduce a way to measure this called Corpus Task Complexity (CTC), showing that tasks needing more complex reasoning become much harder with larger texts. They found that existing models do well on simple tasks but struggle with more complex tasks when the text gets very long. This work highlights the need for better methods to handle complex reasoning over large amounts of text.
What this means in practice
- •For natural language processing teams: Improve evaluation and development of language models by including complex reasoning tasks that scale quadratically with corpus size.
- •For knowledge management teams: Develop better tools for finding contradictions and complex insights in large document collections by understanding task complexity limits.
Authors
Prasann Singhal, Amanda Bertsch, Jacob Steinhardt, Sewon Min
Abstract
Given a large corpus, the questions one might ask can vary -- from "When was the first human heart transplant?" to "What are all the contradictory claims in this literature?" -- but what makes some questions more challenging than others? In this work, we define a notion of Corpus Task Complexity (CTC) that characterizes tasks by how their difficulty grows with corpus size; for instance, a retrieval query only requires a single linear pass over a corpus, while finding contradictions requires checking a quadratically growing set of claim pairs. Observing that prior work has largely only studied tasks whose difficulty grows linearly with corpus size, which we call low CTC tasks, we introduce 10 new tasks belonging to a class of high CTC whose difficulty grows quadratically or more in corpus size. We find that high-CTC tasks not only grow much more challenging on average at longer contexts for LCLMs, they reverse many modeling conclusions drawn solely from low-CTC evaluations. For instance, efficient block-sparse and hybrid attention approaches consistently match full attention performance on low-CTC tasks, but degrade much more on high-CTC tasks. Large-corpus high-CTC reasoning thus remains an open challenge as full attention is too costly to scale, motivating future research on these tasks. We release our code, data, and 22-task suite (CTC-Bench), to facilitate future research in this area.