Tool augmented agents face trust challenges with incorrect evidence
When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents
Artificial IntelligenceCryptography and Security
Summary
Sometimes computer programs that use tools to analyze data get plausible but wrong information from those tools. The authors study how these programs decide to trust or double-check that information. They made a test called ToxicBench to see how often agents accept wrong answers or catch them after checking. Their tests show that these errors can reduce success significantly, and even checking doesn’t always prevent wrong answers from being accepted. They also compared automated scoring to human judgment and found good agreement.
What this means in practice
- •For data analytics teams: Improve reliability checks in systems that augment data analysis with external tool outputs by identifying when to trust or verify evidence.
- •For software engineers: Design better error detection and recovery strategies in automated agents handling numerical and categorical data inputs prone to corruption.
Authors
Zifu Tao, Changqing Yin
Abstract
Tool-augmented data agents rely on tool outputs for analytical decisions. Yet successful execution can return plausible but incorrect evidence, requiring agents to decide whether to trust or verify it. Understanding this failure requires examining both the evidence obtained through checking and the answer ultimately adopted. We introduce ToxicBench to measure checking and adoption under numerical, label, schema, and retrieval errors, pairing clean and poisoned observations over fixed source data. In the 118-task GPT evaluation across three adapters, poisoning lowers task success by 26 to 39 percentage points. Ordinary retries help under one-shot poisoning, whereas repeated poisoning reveals wrong-answer adoption after checking. Controls on three public tables isolate how supplied evidence affects recovery. After freezing the scorer, we compare its judgments with human annotations on 200 trajectories, finding 96% task-success agreement. Human judgments support retry gains over Base and confirm adoption after checking on audited tasks. We release trajectories, versioned scoring, and reference and delivery audits. These findings highlight evidence availability and answer selection as complementary dimensions of agent reliability.