Forensic reconstruction of AI agent logs needs source bindings for accuracy
Correct Answers, Unsupported Findings: Evidence Binding in Forensic Reconstruction of LLM Agent Logs
Cryptography and SecurityArtificial Intelligence
Summary
Figuring out exactly what an AI agent did involves not just picking the right answer but also showing clear proof of where that answer came from. The paper looks at how different pieces of evidence, like tool logs and citation identifiers, help or don’t help in tracing these actions. The authors found that just having the right answer isn’t enough; you need to keep clear links between sources and results to avoid unsupported claims. They tested different methods and showed that adding explicit links between evidence and findings helps improve trustworthiness.
What this means in practice
- •For forensic analysts: Improve accuracy in tracking AI agent actions by using explicit bindings between evidence records and findings.
- •For ai system developers: Enhance AI audit trails by preserving clear links between logs and citations to avoid unsupported conclusions in agent behavior reconstruction.
Authors
Taehyeon Yun, Dongho Kim, Geonwoo Kim, Juyoung Seo, Minseok Hur, Moohong Min
Abstract
Forensic reconstruction of LLM-agent actions requires not only recovering the correct value, but establishing which preserved record supports that finding. Tool logs, generated explanations, and local citation identifiers capture different parts of this evidence, yet a citation identifier does not establish a source unless its binding to a record is preserved. We audit this distinction using 64 mechanically checkable cases from saved AgentDojo Banking executions. Two LLM readers reconstruct source relationships under controlled variations in visible evidence and identifier-to-record bindings. We separately evaluate complete-record agreement, evidence-grounded findings, justified abstention, and unsupported assertions. With original identifiers and no binding table, Sonnet recovered every literal source location but made unsupported citation-source assertions in 26 of 28 cases requiring the missing relation; 22 nevertheless matched the complete reference. Adding explicit bindings improved grounded reconstruction for both readers, whereas identifier renaming alone provided no consistent remedy. A deterministic same-packet comparator correctly resolved the bounded task or abstained throughout. These results show that factual agreement alone is insufficient for evaluating forensic reconstruction of agent logs and motivate preserving explicit record bindings to distinguish supported findings from correct guesses.