Modeling large language model preferences with imperfect decision making
Jointly Modeling Preferences and Imperfect Decision-Making in Large Language Models
Artificial IntelligenceComputation and LanguageComputer Science and Game Theory
Summary
Large language models (LLMs) often make decisions based on multiple goals that may conflict, like helping themselves, helping others, or protecting the environment. The authors found that usual methods to figure out an LLM's goals don’t always predict how the model behaves in complex situations. They created a new approach called the Preference-Behavior Model (PBM) to better understand an LLM’s real priorities and decision mistakes by observing its choices in social scenarios. Their method can tell where an LLM's stated goals and actual actions don’t line up, helping improve how these models act in real interactions.
What this means in practice
- •For ai system designers: Assess LLM preferences and decision inconsistencies to improve AI behavior in multi-agent interactions with conflicting objectives.
- •For interactive ai developers: Tailor LLM responses closer to intended persona priorities by identifying and adjusting misalignments in preference-driven decision-making.
Authors
Axel Abels, Elias Fernández Domingos, Tom Lenaerts
Abstract
Do preferences Large Language Models (LLMs) express in standard elicitation tasks predict their strategic behaviour? LLM agents interacting with humans and other AI systems face potentially conflicting objectives, such as individual welfare, group welfare, and environmental concerns. Prior work characterizes LLM preferences through behavioral-economic and psychological elicitation or strategic games, but rarely tests consistency across settings and prediction in other strategic environments. This requires modeling how competing objectives jointly shape behavior, whereas existing approaches typically measure preference dimensions in isolation. Drawing on utility theory, we introduce the Preference-Behavior Model (PBM), which combines a preference representation with mechanisms capturing imperfect utility maximization. We formulate estimation as an inverse-game problem, jointly inferring preferences and behavioral parameters from observed choices. We evaluate PBM in three social dilemmas that expose conflicting motives: a Public Goods Game, a Collective-Risk Dilemma, and a Common-Pool Resource dilemma. We find that while preferences estimated from established behavioral economic elicitation instruments do not consistently predict behavior in these dilemmas, models fitted in one dilemma retain predictive value in others, suggesting that inferred preferences capture meaningful aspects of LLM decision-making that elicitation instruments may miss. Finally, we assess alignment with persona instructions, finding that inferred preferences broadly reflect intended priorities with specific divergences across dimensions, LLMs, and games being identified by our approach. Our methodology provides a framework for studying how consistently LLM preferences explain behavior, assessing alignment with intended priorities, and revealing trade-offs and behavioral patterns that standard elicitation may miss.