Summary
Large language models (LLMs) are used as social agents by giving them background information to guide their behavior, but small changes in how this information is presented can change their actions unexpectedly. The authors test whether adding different types of extra context to these agents helps make their behavior more consistent and more appropriately sensitive to relevant social details. They create a method to add controlled context and a way to measure how robust the agents are to changes that shouldn't matter. Using a simplified social game, they find that certain kinds of added context lead to more stable and sensible behavior, though results differ between models.
What this means in practice
- •For llm application developers: Build social simulation agents whose behavior is more reliable by adding specific types of background context to prompts.
- •For ai ethics teams: Evaluate and improve the consistency of AI agents’ social decisions by testing sensitivity to context richness in their prompts.
Abstract
Researchers use large language models as agents in social simulations, where agents are prompted with observations and background information intended to shape their actions. Yet LLM agents have been shown to substantially change their behavior under meaning-preserving prompt variations, creating a validation problem for social simulation. One plausible hypothesis is that this brittleness reflects a lack of grounding and can be reduced by enriching the context provided to the agents. Whether richer context improves robustness remains unclear, in part because context richness and robustness are rarely defined in ways that support controlled comparisons. We address the first need with controlled context enrichment, a set of prompt substitution rules that systematically enriches distinct types of context: neutral padding, generic social background, descriptive decision grounding, and normative decision grounding. We address the second by introducing selective robustness as a validation criterion: insensitivity to meaning-preserving prompt variation combined with a directional response to decision-relevant changes in the social world. We demonstrate the framework in a Prisoner's Dilemma-like game, showing how it can be used to identify which forms of context enrichment produce more desirable and robust agent behavior. We find that these forms of enrichment produce distinct behavioral patterns that vary between the two models tested.