Medical reasoning improves by timing evidence lookup with model signals
Taking the PULSE of Medical Reasoning: Step-Level Retrieval Timing for Knowledge-Grounded Inference
Computation and Language
Summary
Medical AI often needs to look up outside information to answer questions correctly, but doing this all the time can slow things down and add mistakes. The authors found that by watching how the AI's internal thinking changes at each step, they can tell when it might need help from external facts. They made a tool called PUlSE that watches these changes and only looks up information when likely needed, keeping accuracy high while cutting down on unnecessary searches. Their method also worked well in a different field, showing it might help many areas beyond medicine.
What this means in practice
- •For medical ai developers: Reduce unnecessary external data queries while keeping high accuracy during stepwise medical reasoning tasks.
- •For legal technology teams: Implement adaptive retrieval in legal reasoning systems to cut down fetching irrelevant documents without losing inference quality.
Authors
Danyang Zhao, Sendong Zhao, Yanrui Du, Haochun Wang, Ming Ma, Wenhao Li, Jingbo Wang, Jiatong Liu, Bing Qin
Abstract
Reliable medical reasoning requires LLMs to dynamically access external biomedical evidence to improve factual trustworthiness and reasoning accuracy. However, indiscriminate retrieval can introduce noise and latency, raising a critical retrieval-timing problem: at each reasoning step, the system must decide whether the model's internal knowledge is reliable or external evidence is needed. We propose PUlSE, a lightweight probing method that uses the model's internal representations as intrinsic signals for step-level retrieval control. Our posterior trajectory analysis reveals that erroneous steps exhibit substantially higher inter-layer volatility than correct ones, suggesting unstable internal knowledge. Based on this finding, PUlSE trains a compact probe on inter-layer geometric features to detect factual errors upon step completion, and uses it as an adaptive trigger for retrieval and step regeneration. Across four medical benchmarks, PUlSE-timed retrieval matches exhaustive per-step retrieval in accuracy while reducing retrieval calls by 72%, and can transfer to an unseen legal domain, indicating potential domain-agnostic generality. Overall, our study provides an initial step toward shifting retrieval-trigger design from surface-level confidence estimates to internal activation dynamics for adaptive access.