Llms show only slight errors when input data seems unlikely
Do LLMs Make More Mistakes If They Do Not Believe the Input Data?
Computation and Language
Summary
Large language models (LLMs) sometimes make mistakes when they process information that seems unlikely or fictional. The authors studied how well LLMs stick to facts when given true, made-up, or fictional data in different languages, focusing on data about local Czech and Slovak facts. They found that these models only slightly reduce their accuracy when handling unlikely inputs, meaning they mostly trust the context they are given. They also found that using a less suitable method to evaluate these mistakes can make the errors seem worse than they are.
What this means in practice
- •For software developers: Improve testing scenarios for LLM-based applications by including slightly implausible or fictional inputs to better understand model errors.
- •For multilingual ai engineers: Develop better evaluation procedures for faithfulness in LLM-generated text across low-resource and localized languages to refine model deployment.
Authors
Peter Kochelka, Aleš Manuel Papáček, Vojtěch Dvořák, Ondřej Dušek
Abstract
Large language models (LLMs) are prone to hallucinating or misinterpreting facts, which impairs their usability in retrieval-augmented generation or data-to-text systems. We analyse how faithfulness of LLMs to provided context depends on how plausible they perceive the context to be (context-memory conflict). To better identify error patterns, we make use of the increased difficulty of non-English and low-resource language text generation and input data based on local knowledge, only partially captured in models' parametric knowledge. We let the models generate text in English, Czech, Slovak and Upper Sorbian from factual (FA), counterfactual (CFA) and fictional (FI) RDF triples containing local Czech and Slovak data. Contrary to our expectations, we observe only a weak context-memory conflict on the human-annotated sample. For Kimi K3 as an LLM judge, which agrees well with human annotations on the sample, counterfactual inputs receive only slightly lower faithfulness scores than factual ones (-0.05 on a 1-5 scale). We also find that a suboptimal choice of LLM judge would lead to overestimating the strength of the context-memory conflict.