Batched language model grading depends on other items in the prompt
Judged in Company: In-Context Batching Makes LLM Verdicts Depend on Other Items
Computation and Language
Summary
When language models judge multiple items together in one prompt to save time or money, their decisions can change based on what other items are included and their order. The authors found that for tasks like spotting harmful comments or grading answers, changing the batch composition affects decisions significantly compared to judging items one at a time. This happens because the model’s internal scores vary with context, and small score shifts can cause decision flips near thresholds. To reduce inconsistency, the authors suggest grading ambiguous items one by one and reusing shared prompt parts to save cost.
What this means in practice
- •For automated content moderators: Detect how batch composition affects harmful comment labels to improve consistency in toxicity filtering systems.
- •For online course platform engineers: Improve automated grading by running ambiguous answers individually to reduce grading variability caused by batch effects.
Authors
Peng Xie, Amr Alanwar
Abstract
To save cost, practitioners often ask a language model to grade many independent items in one prompt. This assumes that verdicts are separable: that the verdict on an item does not depend on the other items in the prompt or on its position among them. Across three judgement tasks (harm in social-media comments, short-answer grading and review star ratings) and five open-weight models with 4-9B parameters, assigning the same 2,000 items to different random batches of 25 changes 11-22% of decisions on harm and grading, 6.5-13.3 points more than renaming the item keys or rewording the instruction, while AUROC against human labels differs from single-item grading only by -0.033 to +0.010; for four models with 14-32B parameters, 6-15% of decisions change, 4.2-9.1 points more. We show that flip rates decompose exactly into components of order, slot and membership, and that under a threshold model they grow with the square root of the variance that the context contributes, so that effects that are small for scores are large for decisions. Most of the latent dissimilarity between partitions comes from the order of the neighbours and the slot. The main effect of which items share the prompt, the part that voting over permutations of a batch cannot remove, accounts for 0.11-0.22 of it, and majority votes over nine permutations still leave 52-60% of the flips. Batched verdicts also differ systematically from single-item verdicts: even the majority of fifteen random batches disagrees with them on 8.4-17.8% of decisions. Editing a single generation shows that both the model's own earlier verdicts and the neighbours' texts carry the effect. On the easy task, star ratings, only 2.9-3.9% of decisions change. We recommend grading ambiguous items one per prompt, recovering most of the cost with a cached prefix, and reporting agreement with single-item grading whenever items are batched.