Memory strategies limit group decisions with large language models
Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions
Artificial IntelligenceComputation and LanguageMultiagent SystemsSocial and Information Networks
Summary
When multiple AI language models work together to make a decision, they need to remember important facts from their discussion. This paper shows that common ways to choose what to remember—picking the rarest facts or the ones most relevant—often lead to mistakes. Sometimes, facts mentioned by many models get too much weight, causing errors. Even keeping the full conversation transcript doesn't always help, as important details still get lost. The authors built tests to understand why these memory choices fail.
What this means in practice
- •For ai development teams: Improve memory selection methods in multi-agent AI systems to prevent loss of critical information during group decision making.
- •For product managers for conversational ai: Design conversation summarization features that avoid over-prioritizing widely known or seemingly relevant facts, enhancing decision reliability.
Authors
Anouar Imel, Imad Charradi, Abderrahman Sghir, Khalil El Khammar
Abstract
When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps. We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, both heuristics fail in predictable places. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration. Rarest-first selection is no better than random selection on average and wins only where the rare facts happen to be the useful ones. In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it. The decision model adds a third failure, a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random with line order fixed, raises accuracy in the classic configuration by 13 to 24 points. In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.