Continuous adversarial training improves honesty in large language models
CAT-alyzing Honesty: Continuous Adversarial Training for Transferable LLM Honesty
Machine LearningComputation and LanguageComputers and Society
Summary
Large language models can sometimes give dishonest answers even after training. The authors developed a new training method called CATCH that helps models learn to be more honest by guiding them away from lying and towards truthful answers. This method works well on different models and tasks and keeps the models useful for other purposes. The training uses only a small set of examples and performs better than other honesty-improving methods.
What this means in practice
- •For ai developers: Enhance honesty of deployed chatbot and AI assistant models by applying CATCH to reduce false claims and unsupported statements in responses.
- •For game designers: Use honesty-trained models to simulate more truthful AI agents in multi-agent games requiring honest behavior, improving realism and user experience.
Authors
Yuan-Chen Chang, Niklas Herbster, Gauthier Gidel, Tommaso Tosato
Abstract
Large language models (LLMs) are becoming prevalent in scenarios where honesty is critical. However, despite alignment training, these models still exhibit dishonest behaviour, such as outputting claims that contradict some knowledge or are not supported by what is accessible in context. We introduce Continuous Adversarial Training for Consistent Honesty (CATCH), inspired by a method originally developed to robustly refuse harmful queries. CATCH trains the model towards an honest answer and away from a dishonest one while maintaining the model's capabilities. We evaluate two architectures (Llama-3.3-70B-Instruct and Qwen3-3.6-27B). We show that trained with only hundreds of targets built from self-reporting scenarios, CATCH learns an honest behaviour that robustly generalizes and transfers across distinct benchmarks. It consistently reduces dishonesty under pressure on MASK, increases confessions in the multi-agent game of Among Us, and improves the discovery rates of the model's hidden behaviours in AuditBench. We show that CATCH outperforms prompting and supervised fine-tuning in improving honesty, and it is on par with state-of-the-art honesty activation steering. However, CATCH surpasses honesty steering in maintaining utility, thanks to a novel choice of utility data.