Large language models detect and reduce false claims by domain training
Domain-Specific Hallucination Detection in Large Language Models
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Large language models sometimes make up information, which is called hallucination. The authors created a system that uses multiple techniques, including a special classifier and uncertainty checks, to spot when these models are likely hallucinating. They tested this system on general and medical topics and found it works best when trained specifically for each area. They also used their detector to help improve another language model, lowering how often it made false claims. This shows the importance of tailoring detection tools to different subject areas.
What this means in practice
- •For natural language processing engineers: Detect false or unfaithful information in model-generated text by combining classification and uncertainty methods for safer text outputs.
- •For biomedical data teams: Improve detection of hallucinated claims in biomedical language models by applying domain-specific fine-tuning techniques.
Authors
Varun Teja Chundru, Debasmita Biswas
Abstract
Large language models generate fluent text that can contain unfaithful claims -- a phenomenon known as hallucination. We present a multi-signal detection pipeline combining fine-tuned DeBERTa-v3 classification, Monte Carlo (MC) Dropout uncertainty quantification, and temperature-scaled calibration for response-level hallucination detection. Evaluated on the HaluEval benchmark, our pipeline achieves F1=0.915 and AUROC=0.977 on general-domain tasks, with per-task F1 scores of 0.97 (QA), 0.96 (Summarization), and 0.82 (Dialogue). MC Dropout inference further improves accuracy to 93.2%. A context ablation study confirms the model performs genuine entailment reasoning rather than exploiting surface patterns, with summarization F1 dropping 24% when knowledge context is removed. Learning curve analysis reveals that 25% of training data captures 77% of full-data performance. Beyond detection, we apply Direct Preference Optimization (DPO) to a Qwen2.5-0.5B generator, reducing its hallucination rate from 85.5% to 37.7% (55.9% relative reduction) as measured by our detector. Cross-domain evaluation on the SciFact biomedical benchmark shows that general-domain training transfers poorly (F1=0.52), motivating domain-specific fine-tuning. PubMedBERT fine-tuned on SciFact achieves F1=0.63 and AUROC=0.81, demonstrating that domain-matched pre-training is the strongest adaptation strategy. Code and models are available at https://github.com/varunteja99/hallucination-detection-nlp