GitHub dataset reveals how AI agent workflows run and fail
GH-AWTrace: A Large-Scale Dataset of GitHub Agentic Workflow Execution Logs
Software Engineering
Summary
GitHub repositories sometimes use AI-powered workflows that can act on their own to do tasks regularly or when something happens. This paper presents a big dataset that collects how these AI workflows actually run across many projects, including detailed logs of their actions and errors. The dataset helps people study what goes wrong, how the AI uses tools, and what the logs tell us about the workflow progress. However, the logs only show parts of the workflows instead of complete stories.
What this means in practice
- •For software engineering teams: Improve error detection and debugging by analyzing detailed AI workflow execution logs from many GitHub repositories.
- •For devops engineers: Monitor automated AI workflows to better understand task progress and handling of external tool failures in real-world settings.
Authors
Jasem Khelifi, Hao Li, Ali Ouni, Mohammed Sayagh, Mohamed Aymen Saied
Abstract
Agentic workflows enable AI agents to perform recurring repository tasks on schedules or in response to repository events. Existing studies examine workflow definitions and outputs, but provide limited evidence of what happens during execution. We present GH-AWTrace, a dataset linking 25,014 verified GitHub Agentic Workflows across 5,051 repositories to 4,112,526 run records and 562,877 nonempty execution log archives. The dataset preserves execution evidence subject to GitHub's log-retention limits and links it to repository metadata and generated workflow configurations. Illustrative cases show how the logs expose tool denials, provider failures, changes in an agent's approach, and delegation among agents. These observations enable research on execution failures, tool use, and the relationship between reported run outcomes and visible task progress. Coverage varies across workflows, and the logs provide partial observations rather than complete agent trajectories. Video: https://youtu.be/KIlfmZhHxtA