Large language models struggle to list complex facts out of order
The Canonical Order Problem: When Large Language Models Are Unreliable Knowledge Bases for Multi-Valued Relations
Computation and Language
Summary
Large language models (LLMs) know a lot of facts and are often used like encyclopedias. But when facts involve multiple related items that aren’t just one answer—like lists of things—LLMs tend to organize them in a fixed order, like alphabetical. The authors found that when you try to get these models to give answers in a different order, their answers become less reliable. This happens because LLMs first retrieve possible answers, then sort them internally in a certain order, before picking the next item.
What this means in practice
- •For knowledge base developers: Improve prompts to align with LLMs’ internal ordering to get more complete multi-item answers.
- •For ai content generation teams: Design systems that respect the LLM’s natural ordering to increase reliability when generating lists of related facts.
Authors
Timo Pierre Schrader, Annemarie Friedrich, Simon Razniewski, Lukas Lange
Abstract
Large language models (LLMs) are increasingly used as knowledge bases (KBs) due to the vast amount of knowledge they acquire during pre-training. While many works focus on extracting single relational triples, most real-world relations are multi-valued and require generating sets of entities. In this paper, we investigate how LLMs represent and generate multi-valued relations. We identify the canonical order problem: The probabilistic distributions inside LLMs organize many multi-valued relations according to a canonical ordering (e.g., alphabetical or chronological). Through mechanistic analysis, we show that set generation in LLMs can be thought of in terms of three phases: (1) retrieval of candidate entities, (2) internal sorting, and (3) selection of the next element. As a result, prompts aiming to construct KBs that deviate from this internal canonical ordering lead to a markedly reduced reliability of LLMs when aiming to generate complete sets for multi-valued relations.