TRACE: TRajectory Attribution for Automated Context Engineering
2026-08-10 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors created TRACE, a system that helps fix mistakes in AI agents by analyzing past interactions where users showed frustration or confusion. Instead of retraining AI models, TRACE looks at different parts of the agent's context like prompts, tools, and knowledge to find the exact source of errors. The system can tell if information needs to be updated or created and works accurately most of the time. Their approach uses past agent behavior as clues to automatically diagnose and fix issues without needing explicit user feedback. They also developed benchmarks to test and verify their method systematically.
AI agentssystem promptscontext sourcestrajectory miningcausal attributionknowledge basesautomated debuggingfeedback loopssimulation benchmarksprompt optimization
Authors
Yikai Zhao, Pradeep Kumar Misra, Saurabh Pandey
Abstract
Production AI agents fail when their context sources -- system prompts, knowledge bases, tool descriptions, and procedural skills -- contain errors or gaps. Current maintenance relies on manual log review and ad-hoc debugging, creating a scalability bottleneck as interaction volume grows. We present TRACE (TRajectory Attribution for Automated Context Engineering), an automated feedback loop that mines historical agent trajectories to diagnose and remediate context failures. Our key insight is that trajectories are rich with implicit dissatisfaction signals -- user corrections, rephrasing, abandonment cues -- that reveal precisely where context sources failed, without explicit feedback collection. Unlike model fine-tuning, TRACE operates on the context layer, enabling rapid iteration without retraining. We make four contributions: (1) a trajectory mining framework that systematically extracts diagnostic information from historical agent executions; (2) multi-component causal attribution that extends textual gradients from monolithic prompt optimization to heterogeneous context sources (skills, knowledge bases, tools, prompts); (3) exploratory verification, where agents actively read context sources to distinguish content gaps requiring CREATE from stale content requiring UPDATE, achieving 96% operation accuracy; and (4) a reusable simulation methodology and verifiable benchmark addressing the absence of open datasets for context debugging, with a six-category fault taxonomy, ground truth annotations, and a cross-layer verification protocol. On 60 dissatisfaction traces spanning three complexity tiers (up to 16 execution nodes), TRACE achieves 72.7% root cause attribution and 82% end-to-end fix effectiveness, showing that over 80% of context-layer failures can be automatically diagnosed and remediated by mining historical trajectories, an overlooked resource in production systems.