Understanding the trade off between learning rules and memorizing facts in AI
Learning Rules, Memorizing Facts: An Information-Theoretic View of Hallucination
Information Theory
Summary
Hallucination in AI models happens when they make up false facts instead of recalling what they learned. This paper looks at how memorizing facts and learning general rules compete within the information an AI can store. The authors use math to show that even when an AI remembers everything it’s seen, it can still make false claims if it has limited capacity. Experiments suggest that the complexity of rules, the amount of training data, and model size all influence how well an AI balances true knowledge and hallucination.
What this means in practice
- •For ai system designers: Predict limits on false fact generation in models by balancing rule learning and memorization given model capacity constraints.
- •For data engineers: Plan training datasets to optimize trade-offs between exposing models to facts and rules, reducing hallucination risks.
A theory result. No direct application yet.
Authors
Gengze Xu, Anxin Guo, Yifei Hu, Xiangyu Wu, Yong Liu, Ziqiao Wang
Abstract
Understanding the relationship between generalization and hallucination requires clarifying the role of memorization. We study this relationship from an information-theoretic perspective through a Selector-Density Conditional Mutual Information (CMI) framework. In the sparse random-fact limit, a uniformly bounded information budget per fact forces a nonvanishing false-positive rate at recall bounded away from zero, even when all facts are observed during training. In a rules-and-facts setting, we decompose model-data mutual information into rule learning, atomic-fact memory, and excess memorization, revealing competition between rule learning and factual memory even without excess memorization. A joint budget bound quantifies the resulting constraint on atomic hallucination under perfect atomic recall. Moreover, an additive Gaussian model shows that maintaining high recall for atomic and composite facts incurs false positives even under optimal data mixing. Finally, controlled relational experiments hint how training exposure, rule complexity, and model capacity shape these trade-offs.