Language agent behavior analyzed through hierarchical trajectory abstractions
Trajectory Abstraction for the Science of Language Agent Behavior
Artificial IntelligenceComputation and Language
Summary
Studying how computer agents use language requires ways to measure their behavior that work across different tasks and models. The authors develop a step-by-step method to create and test layers of behavior patterns, called trajectory abstractions, that summarize agent actions over time. These patterns help generate reliable hypotheses about agent behavior and how it changes under different conditions. They also provide tools to decide if these patterns should be kept, improved, or discarded based on experiments. This approach is different from existing theories or simple behavior descriptions because it links abstract patterns directly back to raw agent actions.
What this means in practice
- •For language model developers: Create generalizable behavioral measurements to compare language models across tasks using the authors' trajectory abstraction method.
- •For human-computer interaction designers: Analyze user-agent language interaction patterns to improve response strategies by applying hierarchical behavior abstractions.
Authors
Tianqiang Yan
Abstract
Scientific studies of language agents need behavioral variables that support hypotheses across tasks and models. We formulate this research problem as learning and testing a hierarchy of trajectory abstractions. A concrete recursive procedure first measures role- and phase-indexed events, proposes temporally constrained relations, and tests their stability across conditions. It then constructs episode-level motif variables from selected relations and repeats the analysis on those variables. Explicit measurement functions connect every abstraction level to the original trajectories. Observations and randomized protocol experiments assess the resulting hypotheses, while comparisons between intervention realizations determine whether an abstraction should be retained, refined, or restricted. We derive a finite-depth bound for accepted reductions, identify protocol effects on fixed abstractions, and characterize realization disagreement and composition of abstraction error. A finite-sample test makes projected intervention consistency operational, and constructed examples illustrate motif construction and abstraction refinement. The formulation distinguishes this experimental approach from semantic taxonomies, qualitative theory induction, and behavior-model recovery. It specifies a proposed research procedure for discovering generalizable behavioral hypotheses, with literature-relative novelty assessed separately from model-relative surprise.