Language model improvements give better confidence in factual claims

Truth in Contrast: Calibrated Factual Evidence from Language Model Likelihoods

Computation and LanguageArtificial Intelligence

Summary

It is hard for language models to know how sure they are when making factual statements. The authors created a way to compare how likely a language model thinks a fact is true versus false, called Contrastive Factual Evidence and its probabilistic version Contrastive Factual Posterior. Their method improves how well the model’s confidence matches reality, especially in high-resource languages. They also found a way to use strong English-language evidence to help models in other languages make better factual predictions without needing extra labels.

What this means in practice

  • For multilingual nlp engineers: Improve confidence estimates for factual claims in language models across many languages, especially by transferring strong English evidence to low-resource languages.
  • For content moderation teams: Enhance reliability of automated fact-checking by better calibrating models’ confidence in detecting true versus false claims.

Authors

Ponhvoan Srey, Jason Lucas, Pittawat Taveekitworachai, Junyoung Lee, Anh Tuan Luu

Abstract

Large Language Models (LLMs) encode substantial factual knowledge. Yet, estimating confidence in factual claims reliably remains challenging. We study likelihood-based factual confidence from the complementary perspectives of discrimination and calibration across multilingual factual statements. Specifically, we instantiate contrastive generative likelihoods as Contrastive Factual Evidence (CFE), which compares the likelihood of the same claim under competing factual and non-factual conditions, and derive its probabilistic counterpart, the Contrastive Factual Posterior (CFP). We show analytically that CFP calibration depends on the differential error between the two surrogate conditionals, allowing errors shared across them to cancel and yield tighter calibration bounds. Empirically, CFP substantially improves calibration while maintaining factual discrimination comparable to its strongest competitor. Nonetheless, discrimination is uniformly weak across methods in lower-resource languages, which simultaneously undermines the utility of the measured calibration. To this end, we interpret a likelihood geometry, within which an evidence-bearing direction is induced, along with an orthogonal evidence-invariant direction. In our investigation, we selectively align target-language claims with stronger English evidence only along the factual-evidence direction, improving discrimination and open-ended factual generation without target-language labels.