Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors
2026-08-17 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how large language models (LLMs) make decisions when they have to explore new options or stick with what they know is good. They created a test called the semantic bandit, where choices have text labels that carry meaning, to see how these labels affect exploration. They found that when labels hint at rewards, LLMs explore less and do better if the hints match the real rewards, but worse if they don't. Also, negative feedback causes the models to explore more than positive feedback of the same size. The authors point out that because LLMs learn from language patterns, their decision-making is biased by the words themselves, which can affect how reliable they are in real tasks.
large language modelsexploration vs exploitationmulti-armed banditsemantic banditpre-traininginductive biasreward structuredecision-making agentsnatural languageenvironmental exploration
Authors
David Eric Austin, Kaheer Suleman, Jackie Chi Kit Cheung
Abstract
Large language models (LLMs) are increasingly deployed as decision-making agents in settings that require sophisticated environmental exploration. However, existing work has raised questions about how LLMs actually balance exploration and exploitation. Unlike classical agents, LLM agents engage with tasks through natural language, exposing them to semantic information with no formal counterpart in the task structure. We introduce the semantic bandit, an extension of the multi-armed bandit setting that explicitly considers the textual labels assigned to actions, and use it to study how semantic priors --- inductive biases arising from associations between language and expected reward learned during pre-training, shape LLM exploration behaviour. We find that semantically informative action labels reduce exploration in favour of exploitation, improving performance when aligned with the reward structure and severely degrading it when misaligned. We further find that negative rewards trigger substantially more exploration than equivalent positive rewards, consistent with an expected-scale bias induced by reward conventions common in pre-training data. Overall, we argue that the use of language to define the environment and rewards introduces unavoidable biases derived from the fact that the model is trained on word co-occurence, with implications for the reliability and robustness of LLM agents in real-world decision-making settings.