Clinical AI agents learn when to stop to avoid costly errors
Trajectory-Derived Confidence for Reliable, Resource-Aware Clinical Text-to-SQL Agents
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
AI systems that translate clinical questions into database queries can make mistakes that are risky and waste computing time. The authors created Sentinel, a system that watches how the AI works step-by-step and decides if it should stop early to avoid wrong answers. Sentinel helps the AI refuse unanswerable questions, stop midway if it detects failure, and withhold uncertain answers, improving accuracy and saving time. This means safer and more efficient use of AI in healthcare data tasks.
What this means in practice
- •For hospital data teams: Increase trustworthiness and efficiency of AI systems converting clinical text into database queries by stopping unreliable reasoning early.
- •For database administrators: Reduce wasted computation on incorrect AI-generated queries in clinical databases by applying a confidence-based stopping system.
Authors
Mincheol Daniel Song, Joshua Ward, Jake Jung, Guang Cheng
Abstract
LLM agents for clinical text-to-SQL applications reason autonomously over multiple steps but cannot assess whether their own reasoning or outputs can be trusted. In high leverage applications such as healthcare, this presents a critical risk where system mistakes can be costly. These reliability failures are also resource failures: an incorrect reasoning trajectory spends computation budget on outputs that must be discarded. We introduce Sentinel, a trajectory-derived, classifier-based confidence layer that analyzes an agent's reasoning, code and database outputs to decide at three points whether to stop: refusing unanswerable questions before the agent runs, halting doomed trajectories mid-run, and withholding untrustworthy answers at delivery. Here, utilizing Chow's rule, we optimize decisions under the EHRSQL shared task's Reliability Score, which penalizes incorrect answers given a utility weighting, and find on the benchmark EHRSQL that Sentinel raises this score from +0.08 to +0.24 when mistakes have a low utility weighting, well above the +0.03 earned by refusing every question, with delivered-answer accuracy rising from 54% to 69% as coverage falls from 84% to 55%. At higher stakes the agents we test rarely answer reliably enough to deliver, and Sentinel detects this on its own, abstaining to that same +0.03 where the unmonitored agent scores -3.02. The same stopping decisions cut computation: at low stakes, where the system still answers, Sentinel eliminates 13-28% of agent steps at little or no reliability cost.