Ai understanding arises from many interacting parts not one simple mechanism
A Polyphonic Conception of AI Understanding
Artificial IntelligenceComputation and Language
Summary
AI models like large language models don’t understand things through a single clear part inside them. Instead, their outputs come from many different parts working together in complex and overlapping ways. This makes it hard to say exactly what the AI understands or trusts. The authors suggest thinking about AI understanding as depending on reliable internal circuits controlling the output, which helps us decide when to trust AI results more carefully.
What this means in practice
- •For legal teams: Decide when to trust AI advice in legal decisions by evaluating the reliability of internal AI components controlling outputs.
- •For healthcare providers: Assess AI diagnostic tools by identifying which internal AI circuits consistently generate trustworthy medical outputs.
A position paper. It proposes an approach and reports no results.
Authors
Matthieu Queloz, Pierre Beckmann
Abstract
When a doctor, a judge, or an engineer must decide whether to trust an AI model's output, they cannot avoid asking what the model understands. Purely mathematical or statistical descriptions struggle to distinguish trustworthy from untrustworthy outputs without reintroducing the question of AI understanding in all but name. Yet the question is ill-framed as it stands, because the inherited concept operates within a monophonic paradigm: the idea that a cognitive system's understanding of something must be localised to a single mechanism underpinning all the capacities conferred by such understanding. Drawing on a wide range of mechanistic evidence, we show that LLMs are pervasively polyphonic: outputs emerge from coalitions of parallel mechanisms of uneven reliability, which variously complement, duplicate, or drown out one another, with several coalitions sufficing for a task without any one being indispensable. Polyphony not only complicates attributions of understanding, but renders monophonic inference patterns hazardous. In response, we develop a conception of understanding fit for polyphonic AI. It centres on sound circuitry that is reliably and correctly recruited and in control of outputs. Attributions of understanding thereby become tractable claims about internal organisation, and can do the work of guiding trust in AI.