Persian medical models gain faster claim-level hallucination detection
Single-Pass Uncertainty Heads for Claim-Level Hallucination Detection in Persian Medical Language Models
Computation and Language
Summary
Language models sometimes generate incorrect medical information, which can be dangerous. The authors worked on detecting these 'hallucinations' for Persian medical language models without needing slow repeated checks. They built new tools that quickly estimate uncertainty in answers to spot possible errors in one pass. Their experiments showed promising results even though the datasets were small and automatically labeled.
What this means in practice
- •For medical ai developers: Integrate single-pass uncertainty heads to flag potentially incorrect claims in Persian medical chatbots without costly multiple outputs generation.
- •For healthcare it teams: Implement faster hallucination detection directly in Persian medical language model pipelines, improving reliability of automated medical assistance tools.
Tested on one dataset.
Authors
Mehrdad Ghassabi, Pedram Rostami, Hamidreza Baradaran Kashani, Sadra Hakim, Audrina Ebrahimi
Abstract
Hallucination detection is particularly important for medical language models, but repeated-sampling approaches are expensive and existing uncertainty-head resources do not directly transfer to a new backbone and language. We adapt the LLM Uncertainty Head (LUH) framework to Aya-Expanse-8B-based Persian medical models, using Gaokerena-V and Gaokerena-R as two previously developed backbones. We first examine response variability on a 168-question Iranian medical entrance examination and observe substantially lower five-run consistency for Gaokerena-V than for Aya-Expanse-8B, whereas Gaokerena-R is comparable to Aya-Expanse-8B. We then construct two paired claim-level hallucination datasets directly in Persian, containing 1,600 responses for each backbone, and train lightweight claim-level heads on frozen backbone attention maps and token probabilities. On held-out test splits, the heads obtain PR-AUCs of 0.4820 and 0.4652, corresponding to 2.30 and 2.66 times their respective random baselines, and ROC-AUCs of 0.7852 and 0.7810. The heads require neither retrieval nor repeated sampling at inference time. These results provide an initial study of single-pass claim-level uncertainty estimation for Persian medical language models; the test splits are small and the labels are automatically generated.