Structured tracking reveals what ai coding agents actually do
Soei: Structured Observability of Evidence and Intent for Multi-Agent AI Software Engineering
Software EngineeringArtificial IntelligenceMultiagent Systems
Summary
When AI coding agents write software over multiple sessions, it can be hard to tell what changes they actually made, who did them, or whether the work is finished. The authors studied how to track these actions precisely using a tool called Agenttrail, analyzing real coding sessions and finding gaps like unfinished checks and missing proofs for completed tasks. They then proposed a new system, Soei, to record evidence and intentions with secure and verifiable logs so users can trust what was done and who did it. This approach helps developers understand and manage AI collaborators better.
What this means in practice
- •For software development teams: Track and verify multi-session AI coding actions with secure logs to confirm who made changes and whether tasks are truly finished.
- •For security analysts: Detect and prevent unauthorized code changes or leaks by monitoring AI agent activity and sensitive credential handling in development sessions.
Authors
Mohammed Nayeem Khan, K S Sarun Kumar
Abstract
AI coding agents now do long, multi-step and multi-session software work with little supervision. When a developer comes back to that work, the useful question is not what the model said but what was actually done, by whom, and whether it is finished. We study this in three parts. First, we verify Agenttrail, an open-source local observability tool for coding agents, against its source at commit e4ba2da: nine documented constants and mechanisms hold at file-and-line level, two event pipelines are never unified, and its Map service has an unauthenticated local mutation surface. Second, we run a prototype of our system on 45 real coding-agent sessions (102,233 records, 20,577 tool executions, 2,336 human-to-agent episodes). With a blind-audited classifier (precision 83.3% and 96.7%) we estimate that 23.4% (95% CI 17.1 to 35.7) of code-editing episodes end with no independent check after the last edit, and that 57.8% of to-dos the agent marked complete after editing had no proof in between. A 10-minute timing heuristic for session hand-offs recovers 6 of the 24 session pairs that content lineage links. 42% of sessions run longer than 24 hours, 48.1% of transcript bytes lie past a 2 MiB replay bound, and 1,326 records in 16 sessions contain credential-like strings. Third, we propose Soei (Structured Observability of Evidence and Intent): an acknowledgement-aware event schema, a hash-chained local ledger, a W3C PROV-aligned graph, calibrated inference for attribution and lineage, and a verification layer that keeps a claim of completion apart from proof. On the same corpus the ledger is 20% of raw size, holds no content or credentials, verifies at about 126k events/s, detects tampering and replays deterministically. We position the work against concurrent 2026 proposals and define autonomy levels with evidence gates and harness metrics baselined on our data.