Agent behavior maps identify AI actions without running them
Agent Behavior Maps: Locating Agents by What They Would Do, Without Running Them
Machine LearningArtificial IntelligenceComputation and Language
Summary
Different AI agents can solve the same tasks successfully but take different steps to do so. The authors introduce a way to compare AI agents by looking at what actions they would likely take on shared example tasks, without needing to actually run these agents in the environment. They show their method can predict how similar different agents are and how well they might perform in various settings. This helps understand agent behavior and compare models efficiently.
What this means in practice
- •For ai model engineers: Identify similar AI models and predict their performance without costly environment simulations.
- •For game ai developers: Compare and cluster AI behaviors in simulated game environments to improve agent design.
Authors
Shota Fujikawa, Takuya Tamura, Kosuke Akimoto, Daichi Haraguchi, Masafumi Oyamada
Abstract
LLM agents with similar success rates can follow different action sequences. An Agent Behavior Map represents LLM-harness combinations using centered action log-likelihoods on shared reference trajectories, without requiring new environment rollouts from the target agent. We derive error bounds relating squared coordinate distances to trajectory KL under explicit assumptions, and across eight cross-scored models the distances track the directly measured trajectory KL in rank order. Across 100 LLMs and four environments, ALFWorld-based coordinates support performance prediction and identify behavioral similarities. In ALFWorld, these similarities persist after controlling for success rates and model size, including across model families. As an extension, harness-component displacements approximately compose, with accuracy varying across components and environments.