Agentic LLM auditing reveals hidden shortcuts behind scientific predictions
Auditing Tool Traces in Agentic LLMs: Source Localization and Counterfactual Dependence Testing
Artificial IntelligenceComputation and Language
Summary
Large language models (LLMs) that use external tools are meant to make scientific predictions by carefully using evidence. The authors found that these models might appear to perform well because they pick up hidden shortcuts or clues in the data, not because they understand the science. They created a two-step audit process to find these tricks and check if the model really depends on them. Their tests in biology showed the model’s performance dropped to chance when those shortcuts were removed, meaning it wasn’t truly reasoning. This means audits like theirs are important before trusting AI results in science.
What this means in practice
- •For biotech software engineers: Detect hidden shortcut signals in tool-augmented LLM outputs to ensure authentic scientific reasoning in biology tasks.
- •For ai safety teams: Use source-level audits to verify that agentic LLMs depend on valid evidence rather than dataset artifacts when deployed.
Authors
Junhao Liu, Tianxiao Li, Yongxian Wu, Ziheng Duan, Ren Li, Jing Zhang, Martin Renqiang Min
Abstract
Agentic LLMs increasingly support scientific prediction by grounding decisions in external evidence. In biology, they promise accurate, interpretable predictions for important problems by integrating complex experimental and computational evidence. However, serialized tool results can contain hidden label shortcuts, such as dataset identifiers or template artifacts, that create apparent success without biological reasoning. To address this risk, we introduce a two-stage audit. Stage 1, Per-Tool TF-IDF Decomposition (PT2D), localizes label-predictive patterns to individual tool results while avoiding signal dilution in the concatenated tool trace. Stage 2, Token-Class Redaction, counterfactually tests whether the frozen LLM depends on the localized shortcut. We apply this audit to TCR-pMHC binding, where tool augmentation substantially outperforms a no-evidence baseline. At face value, this suggests that the LLM has learned to use scientific evidence. The audit overturns this interpretation: PT2D localizes the strongest signal to identifiers in one tool result, and redacting them reduces performance to chance. These findings show how hidden label shortcuts can masquerade as scientific reasoning or discovery, motivating source-level audits before benchmark gains are trusted.