Detecting lies in large language models using persona and knowledge cues

The Geometry of Lies: Knowledge and Persona Directions for Detecting Deception in Large Language Models

Artificial Intelligence

Summary

It can be hard to tell when AI language models are lying, especially when lies look like honest answers. The authors tested ways to find clues about lying using role-playing personas and knowledge-based questions in four big AI models. They found that some methods work better on some models than others, and that clues based on the wording of answers aren’t always reliable. Combining clues about persona and knowledge can improve lie detection, but there’s still no perfect method.

What this means in practice

  • For ai safety teams: Identify deceptive outputs from large language models more reliably by combining persona and knowledge-based detection methods.
  • For chatbot developers: Improve chatbot trustworthiness by applying persona and knowledge cues to detect when a model might be giving false information.

Authors

Md. Abdullah Al Zami, Suraiya Shanaz, Mahdi Hasan, Shoaib Ahmed Dipu, Farig Yousuf Sadeque, Md. Sabbir Ahmed

Abstract

Linear probes are the cheapest LLM lie detectors but often fail to transfer across kinds of lie. Label-free directions from honest and deceptive role-play are untested at 70B scale and on lies about knowledge, where a lie and an honest denial can be identical. We widen a published persona bank with roles that deny knowledge they have, plus honest counterparts. At each model's middle layer the resulting persona direction ranks lies across Liars' Bench better than single-dataset probes on three of its four models; it fails on Qwen, even when Qwen stays in role far more often. On Llama-3.3-70B-Instruct it beats directions from a random persona bank, but its gain over the published direction is within the range of random widenings. On lies about knowledge, persona directions mainly track a reply's wording: where lies and honest denials share their words, our three main contrasts stay inside the random-direction band on Llama and Qwen and reach only its edge on Gemma, though on Mistral one clears it. A knowledge direction contrasts questions the model can answer with ones nobody can, each followed by the same scripted denial. It separates lies from identical honest denials on two models and narrowly on a third, but not on the fourth; its other results partly depend on the benchmark's wording. On the Gender-Secret fine-tune it clears the band by a few hundredths on Llama and fails on Mistral. The sum of the two directions ranks lies better than a supervised probe at the same depth on all four models (on one, both near chance) but is level with the persona direction on Llama's average and behind that probe on held-out lies. More varied role-play questions do not help the persona direction on Liars' Bench.