Long context AI models show different biases in using evidence positions
Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
When AI models answer questions using long texts, where important information appears in the text affects how well the models use it. The authors studied different types of models and found that some, like Transformers, pay more attention to information at the beginning of the text, while others do not. Both types tended to favor evidence near the end as well, but how they use the information depends on the model design and how the question is asked. This means that the way a model mixes sequences influences how it focuses on different parts of long texts, but it’s not the only factor.
What this means in practice
- •For ai system developers: Design long-context question answering systems that adjust evidence placement to improve accuracy depending on model type and prompt setup.
- •For chatbot builders: Tune chatbot prompts and input ordering to reduce position bias effects and achieve better answer relevance in long conversations.
Authors
Aman Behera, Namit Solanki, Mehul Anand
Abstract
Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.