Visual presentation reduces bias in language model action choices
Procedural Exposure Bias in Skill-Augmented Language Model Agents
Computer Vision and Pattern Recognition
Summary
Language models that use skills often tend to prefer actions they have seen many times before, even if the situation doesn't call for them—this is called Procedural Exposure Bias (PEB). The authors found that when skills are shown visually rather than as text, this bias is much weaker. In other words, models are less likely to overuse repeated actions if the instructions are displayed visually. This difference happens because the model processes visual information differently, making the repeated action cues less influential on what the model decides to do. The findings help explain how the way skills are presented affects how language models make decisions.
What this means in practice
- •For language model developers: Design skill interfaces with visual layouts to reduce biased overuse of repeated actions in agent behavior.
- •For conversational ai engineers: Improve task success by incorporating visual skill presentations into language-model-based agents in interactive environments.
Authors
Qing Yang, Yuqi Xiong, Pengcheng Huang, Zhipeng Xu, Zhenghao Liu, Zulong Chen, Yu Gu, Ge Yu
Abstract
External skills provide language-model agents with reusable procedural knowledge, yet how models use such knowledge after it is supplied remains poorly understood. We identify Procedural Exposure Bias (PEB), the tendency of language models to increasingly favor an action as its associated procedural pattern is repeatedly exposed in a skill, even when the interaction state and action relevance remain unchanged. We find that PEB strongly depends on how procedural knowledge is represented: presenting the same content visually substantially attenuates exposure-induced shifts in action preference relative to textual presentation. Further analysis shows that this attenuation primarily arises from visual encoding rather than structured layout. Although repeated procedural signals remain recoverable from Visual representations, they exert a weaker influence on downstream action prediction, distinguishing signal accessibility from its use as action evidence. This effect also extends to executable action selection, where increasing exposure more strongly favors the corresponding action under Text than Visual. Layer-wise analyses show that exposure-dependent action evidence becomes more pronounced toward the model output, with stronger evidence under Text, while activation interventions reveal architecture-specific roles of internal components in transmitting or counteracting the preference shift. Complete-agent evaluations across ALFWorld, ScienceWorld, and WebShop further show that Visual skills achieve higher macro-average task success than Text skills for all four evaluated models. These findings establish skill representation as an important factor governing how procedural knowledge is translated into agent decisions. All code is available at https://github.com/NEUIR/Vis-Skill.