Clarifying what makes AI count as an agent and how to measure it
Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks
Artificial IntelligenceMultiagent Systems
Summary
It can be hard to say exactly what an AI agent is because different researchers use the word differently. The authors looked at five key features that AI agents might have, like how they interact with their world, learn, act on their own, aim for goals, and keep consistent over time. They gathered many ways these features have been measured before and made a resource that collects these measures together. This can help people compare and test AI agents more clearly and fairly.
What this means in practice
- •For ai developers: Use structured criteria and benchmarks to consistently evaluate different AI agents’ capabilities during development.
- •For robotics engineers: Compare the autonomy and goal-directed behavior of robots using standardized metrics to guide system improvements.
A survey. It maps existing work.
Authors
Mia Lassiter, Brinnae Bent
Abstract
The term agent in artificial intelligence lacks a standard definition, complicating the evaluation, comparison, and reproducibility of AI agent research. We address this ambiguity through a survey organized around five dimensions of agenticness: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. For each dimension, we examine how the underlying capability has been conceptualized across prior work and synthesize the metrics, benchmarks, and evaluation frameworks used to assess it. This review provides a structured account of the current landscape of agent evaluation, highlighting both established approaches and areas where evaluation remains limited or inconsistent. We additionally introduce the Agent Compendium, a public-facing digital resource that organizes and extends the evaluation methods identified through this review. Together, the survey and compendium provide a common structure for evaluating and comparing agent capabilities across AI systems, supporting more reproducible research, clearer communication, and more systematic study of artificial agents.