The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
2026-07-14 • Computation and Language
Computation and Language
AI summaryⓘ
The authors studied how large language models (LLMs) respond when given extra, often irrelevant context alongside their tasks. They found that while overall accuracy seems stable, individual predictions can change a lot, even when the extra context is just random gibberish. This effect varies depending on factors like the type and length of context, and the model's training stage. Their work shows it's important to check model reliability on each example, not just overall performance.
large language modelscontext relevancemodel robustnessbenchmark accuracyper-example evaluationsemantically meaningless contextmodel instabilitytest-time computemodel development stagesprediction variability
Authors
Yanzhe Zhang, Sanmi Koyejo, Diyi Yang
Abstract
As large language models (LLMs) grow more capable, they are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context. In a controlled setting, we find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy. This aggregate stability, however, masks significant per-example instability. Even semantically meaningless pseudo-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others. This two-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model-specific. We further show that this instability is modulated by context type, context length, test-time compute, and model development stage. Together, our findings reveal context-induced tail risks concealed by aggregate accuracy, motivating per-example reliability evaluation of language models.