Using Grounded Theory for Agent Behavior Analysis at Scale
2026-08-31 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors created AutoTraceGT, a tool that uses a social science method called Grounded Theory to analyze how agents behave by studying lots of their action paths. This method helps identify different behavior patterns without relying on predefined rules. In tests, their tool found most known failure types and even discovered new ones, matching expert understanding. It also helped predict failures better than some AI language models. Overall, the authors show that applying this qualitative method can help researchers better understand agent behaviors at scale.
Grounded Theoryagent trajectoriesbehavioral taxonomyfailure modesopen codingaxial codingtheoretical codingzero-shot learningfew-shot learningfailure prediction
Authors
Zhuoran Lu, Yangyang Yu, Zhuoyan Li, Yibo Meng, Nan Jiang, Chengxi Zang, Jie Gao, Ziang Xiao
Abstract
Understanding agent behavior requires methods that scale to thousands of trajectories and surface new patterns in long, often unfamiliar tasks where pre-built classifiers fall short. We propose to bring grounded theory into agent trajectory analysis: a six-decade-old qualitative method from the social sciences, with a principled saturation criterion and an auditable trail from data to theory. We propose AutoTraceGT (Automated Trace analysis through Grounded Theory), the first multi-agent pipeline that automates grounded theory on agent trajectories. It iteratively performs open, axial, and theoretical coding until saturation, producing a behavioral taxonomy tailored to each task. Across six trajectory corpora, AutoTraceGT produces codebooks that recover 73-91 percent of the failure modes in human-annotated taxonomies and surface additional patterns that those taxonomies miss. The emergent theoretical narrative aligns with prior expert accounts. Used as a deductive feature space, the codebook outperforms zero-shot and few-shot LLM baselines on downstream failure prediction. These results suggest Grounded Theory offers a scalable analytic tool for ML researchers and agent developers studying what agents actually do.