Language models judge facts less well outside english and this closes the gap
Fair Fact-Checking: Closing the Cross-Lingual Gap in LLM Factual Judgement with RoSh
Computation and LanguageArtificial Intelligence
Summary
Fact-checking with AI models is less reliable in languages other than English, especially for smaller models and languages like Arabic. The authors found that while models often know the correct answer, they fail to express it properly in many languages. They propose RoSh, a method that adjusts the model’s internal processing without retraining, greatly improving fact-checking accuracy in multiple languages. This method closes most of the performance gap between English and other languages, making fact-checking fairer across languages.
What this means in practice
- •For social media platforms: Improve multilingual fact-checking accuracy by integrating RoSh to better identify misinformation in various languages.
- •For multilingual chatbot developers: Enhance the reliability of factual judgments in chatbots across different languages without costly retraining.
Authors
Muhammad Ahmad, Fatemeh Seyedin, Adrian Weller, Dongwon Lee, Mahmoudreza Babaei
Abstract
Misinformation on social media remains a critical problem, and more and more people settle it by asking a language model instead of a fact checker. Whether models judge such claims reliably is debated; whether they judge them equally well in every language people ask in has gone almost unasked. We test eight models from five families, 3B to 70B, on 1,500 encyclopedic factual claims that exist in identical form in eight languages. English is judged better than every other language on every model, and the gap is widest on the smallest ones, where Llama-3B on Arabic is no better than guessing. Existing remedies retrain on more multilingual data or fit an unconstrained map between language representations, and neither asks whether the model already holds the answer and simply fails to say it. It largely does: a linear probe recovers the truth from the very activations the model fails to express. We propose RoSh, a per-language shift and rotation of the residual stream, computed in closed form at three layers, with no training and no weight modified. It improves every model and closes 75% of the gap on average, helping most where the model was worst: Arabic on Llama-3B goes from chance to nearly the English level, and a fifth fewer of the claims answered correctly in English are lost in translation. What remains is no longer a read-out failure: afterwards the head recovers as much of what is encoded outside English as it does in English. An unconstrained map fitted on the same pairs falls below the untouched baseline, so the orthogonality constraint is doing the work, and every model clears a scrambled-correspondence control and ten further controls. On the two benchmarks of the closest inference-time method, latent-space intervention, run with its own data and metric code, RoSh's gains are five to thirteen times larger.