Large language models show strong sensitivity to input order in analysis
Does Every Instance Count Equally? Order Effects in LLM Multi-Instance Processing
Artificial Intelligence
Summary
When people ask large language models (LLMs) to look at many separate pieces of information and give one final answer, the order of that information can change the answer a lot. The researchers tested 15 models on different datasets and found the results could change by almost 90% depending on how the inputs were arranged. Models usually did better when the most common answer appeared near the start. This means that LLMs do not treat all pieces of information equally when combining them, which can cause bias in their responses.
What this means in practice
- •For product review analysts: Improve reliability of sentiment summaries by testing model outputs against different review orderings to detect potential biases.
- •For social media monitoring teams: Assess how message ordering affects topic or sentiment aggregation to ensure consistent trend detection from social media posts.
Authors
Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-Collados
Abstract
Users increasingly rely on large language models (LLMs) to analyse collections of independent inputs, such as product reviews or social media posts, and return a single aggregated answer. This setting requires multi-instance processing (MIP), where a model combines information from multiple instances into one prediction. We study a task in which the model must infer a binary label for each instance, and identify the most frequent label among the full instance set. The correct answer should be invariant to instance order. By evaluating fifteen LLMs across five datasets and seven instance order patterns, with each instance set held fixed across patterns, we find substantial order sensitivity. Accuracy can differ by about 90 percentage points between the best and worst order patterns, with models generally more accurate when majority-label instances appear at the start than when they appear in the middle or at the end. For subsets of models, available reasoning traces and gradient-based attribution reveal uneven emphasis across positions. Traces disproportionately mention early instances, while attribution scores are generally higher near the beginning and end than in the middle. These results show that instance order can systematically bias aggregated predictions, highlighting the need to evaluate MIP across different input arrangements.