Calibrating Small Language Models for Claim Check-Worthiness Detection
2026-08-31 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors address a problem where running big language models (LLMs) on every claim to check if it should be verified is too slow and expensive, while smaller models are less accurate. They introduce NN-PPI, a new method that improves predictions by adjusting model outputs after they are made, without needing to retrain the model. Their approach significantly boosts the accuracy of smaller models, making them perform like larger, more expensive models. This helps make claim check-worthiness detection cheaper and faster without losing quality.
claim check-worthinessfact-checkinglarge language models (LLMs)small language models (SLMs)Prediction-Powered Inference (PPI)post-hoc calibrationweighted F1 scoremodel fine-tuningcost-efficiencyinference time
Authors
Pratuat Amatya, Venktesh Viswanathan, Vinay Setty
Abstract
Assessing claim check-worthiness is an essential first step in automated fact-checking pipelines. This work is motivated by a real deployment challenge at an early-stage startup: running large language models (LLMs) over every incoming claim is cost- and latency-prohibitive, yet smaller models sacrifice accuracy. We propose NN-PPI, a pointwise extension of Prediction-Powered Inference (PPI) that calibrates model predictions at inference time as a lightweight post-hoc layer, without re-training the underlying model. NN-PPI achieves weighted F1 gains ranging from 12% to 33.80% depending on the size and performance of the baseline model, bringing SLMs on par with larger LLMs. Beyond few-shot SLMs, NN-PPI further improves a production-deployed fine-tuned model, demonstrating that residual calibration is complementary to supervised fine-tuning. By recovering LLM-level accuracy from models that are an order of magnitude cheaper to serve, it makes accurate check-worthiness detection substantially cheaper to operate at scale. Our code and data can be found at https://anonymous.4open.science/r/arr-claim-worthiness-F237.