Tool helps track and analyze long AI agent task transcripts
Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations
Artificial Intelligence
Summary
It is hard to understand what happens when AI agents work on very long and complex tasks, because their records can be hundreds of pages long. The authors created Transect, a tool that helps people explore and follow the behavior of AI agents step-by-step. Transect links different actions, labels, and parts of the conversation so reviewers can quickly find interesting moments and check their interpretations. This tool makes analyzing long AI runs easier, more repeatable, and clearer.
What this means in practice
- •For ai evaluation teams: Use Transect to navigate and analyze extensive AI agent task transcripts to identify behavior patterns and improve evaluation clarity.
- •For software engineers: Integrate Transect’s timeline-based reports into AI development pipelines to audit and debug complex multi-agent interactions.
Authors
Toby D. Pilditch, Konstantinos Voudouris, Alexandra Abbas, Cozmin Ududec
Abstract
Frontier AI evaluations increasingly use open-ended, agentic, long-horizon tasks whose transcripts can span hundreds of pages of outputs and actions from complex multi-agent networks. The observability envelop-the range of what evaluators can reliably infer about an agent's behaviours-is therefore narrowing. Language model assistants can help classify and interpret agent behaviour but also afford human evaluators significant analytical degrees of freedom, threatening the reproducibility and auditability of language-model-based transcript analysis. Transect is an open source package built on Inspect Scout to help evaluators understand how a long agent run unfolded, identify behaviour worth investigating, and check interpretations against the transcript. Users specify task context and behavioural vocabulary in a reusable evaluation-family configuration, with judge models and analysis settings supplied separately. Transect's navigable reports align recorded events, token use, sub-agent activity, and model-generated behavioural labels on a common turn-based timeline. Reviewers can quickly grasp a run's narrative, trace any label or event to its source turns, and export the underlying data tables for cross-run analysis. We demonstrate the workflow on an AI R&D evaluation that generated almost 13 million tokens, dividing the agents' work into behavioural phases aligned with research-skill classifications, sub-agent delegations and interactions, and token use. The combined view shows a focus on operational work and manuscript production, with little evidence of a sustained hypothesis generation stage-arguably a necessary component for high-quality scientific outputs. Transect's flexible, customisable transcript-analysis pipeline will enable evaluators to keep pace with longer, more complex, more frequent AI evaluations while supporting scientific rigour, transparency, and reproducibility.