Altas improves clinical language model answers using question-by-question routing
Route, Don't Fix: Regime-Dependent Decoding Correction and a Trajectory-Gated Router for Reliable Clinical LLM Answer Selection
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Large language models can give wrong answers when asked medical questions, which is risky. The authors designed ALTAS, a method that checks how confident the model is when it answers each question and decides whether to trust the first answer or apply a correction process. This improves accuracy a lot on tricky questions while keeping medical test results reliable. ALTAS works without needing extra training and adds only a small delay when used.
What this means in practice
- •For clinical ai developers: Use ALTAS to improve reliability of language models answering medical questions without extra training or infrastructure changes.
- •For natural language processing engineers: Implement ALTAS gating to balance answer correction and decoding speed in multiple-choice medical language model applications.
Authors
Zeyu Dong, Benjamin Wang, Joyee W. Jin
Abstract
Large language models (LLMs) are often deemed unsafe for clinical question answering because of their tendency to hallucinate. Retrieval augmentation, fine-tuning, and external verifiers require new infrastructure that clinical governance must approve and may add latency or extra model calls. Inference-time correction uses the model's internal logit signals, but a fixed transformation need not suit every question. A corrector that improves accuracy by about ten percentage points on a truthfulness stress test yields negligible gains on clinical multiple-choice benchmarks, where instruction tuning concentrates output probability on one answer and leaves low terminal entropy. We introduce ALTAS, which reads terminal entropy and late-layer linearity ($R^2$) from one forward pass to choose per question between greedy decoding and late-layer trajectory correction. No classifier, probe, or head is trained; the router operates on candidate-answer logits and adds 6.5% latency overhead. Applied to every question, the correction improves TruthfulQA over greedy at 3B and 8B by 11.4 and 10.0 percentage points, respectively ($p<10^{-10}$). Gated per question, ALTAS retains gains of 8.3 to 9.5 percentage points while keeping MedQA, PubMedQA, and MedHallu within a one-percentage-point do-no-harm band, with no statistically significant differences from greedy. The method passes verification sweeps over frozen thresholds, the scoring rule, and the domain label.