Bayesian reasoning explains how agents show signs of intelligence
Bayesian Intelligence from the Outside
Artificial Intelligence
Summary
Figuring out if a machine or AI is truly 'intelligent' by watching what it does is hard. The authors propose a way to see if an agent like a language model acts like a Bayesian reasoner, meaning it updates its beliefs based on new information and honestly reports uncertainty. They show that if an agent’s answers don’t fully contradict each other, then its behavior fits this smart reasoning pattern. But combining uncertain answers from different agents is tricky unless they provide complete beliefs about the world. This work helps us understand when an agent’s behavior really counts as intelligent.
What this means in practice
- •For ai system designers: Check if an AI agent’s answers can be explained by Bayesian belief updating to assess its rationality.
- •For data fusion engineers: Know the limits when combining uncertain reports from different intelligent agents without full belief information.
A theory result. No direct application yet.
Authors
Alex Smolin, Bryan Wilder
Abstract
Inferring intelligence from observable behavior is a foundational challenge in artificial intelligence. We develop a theory of Bayesian intelligence for agents such as language models. Each prompt induces a possibly imperfect internal experiment; the agent updates a full-support prior by Bayes' rule and faithfully reports its posterior over the possible answers to the question. Repetitions draw fresh, independent outcomes from the same unobserved experiment at one fixed state. We show that the agent's behavior admits this explanation if and only if its reports are not fully contradictory, i.e., some state remains possible under every report across all prompts. Report frequencies and the sizes of positive probabilities impose no further restrictions. We further propose and characterize the behavioral implications of an intelligence order that makes the behavior of two agents consistent with one agent having access to a more informative experiment: there should exist a coupling of report distributions such that the more informative agent's report excludes every answer excluded by its counterpart. Finally, we show the difficulty of aggregating coarse reports from intelligent agents: unless the agent reports a belief about the complete state of the world, the optimal aggregation can assign arbitrary weights to states that have not been excluded. These results provide a basis for understanding when agents' behavior is intelligent and highlight the difficulty of rejecting Bayesian rationality.