Language models learn to balance truthfulness and creativity in writing
HARPO: Hallucination-Aware Reinforcement Learning for Faithful and Creative Language Generation
Computation and Language
Summary
Large language models often make up false information, which makes them less reliable for tasks requiring accurate knowledge. The authors introduce HARPO, a new training method that helps models write both creatively and truthfully by using a special system to detect hallucinations and reward good writing only when it’s accurate. Their experiments show that HARPO reduces false information while making writing better and more creative at the same time.
What this means in practice
- •For conversational ai developers: Create chatbots with less false information while maintaining engaging and creative responses.
- •For knowledge base maintainers: Improve automated language generation for knowledge-intensive documents by reducing hallucinated content.
Authors
Tiezheng Yu, Yuxin Jiang, Jinpeng Li, Shuning Sun, Fei Mi, Haoli Bai, Lifeng Shang
Abstract
Large Language Models (LLMs) are prone to generating hallucinated content, which compromises their reliability in knowledge-intensive tasks. To address this challenge without sacrificing creativity, we propose HARPO, a reinforcement learning framework designed to jointly optimize faithfulness and creativity. HARPO incorporates a Hallucination-Aware Generative Reward Model (HA-GRM), trained via verifiable feedback, to assess both faithfulness and writing quality. A Selective Activation Mechanism (SAM) activates writing rewards only for outputs judged hallucination-free by HA-GRM, while a data curriculum progressively shifts training from creative writing to hallucination-centric tasks. On RAGTruth, our Qwen3-4B-based HA-GRM achieves a response-level F1 score of 78.08%, compared with 66.37% for the supervised fine-tuning baseline. Experiments on Qwen2.5 and Qwen3 models from 1.7B to 8B parameters show improvements in both faithful generation and writing quality. On Qwen3-4B, HARPO reduces the HA-GRM-judged hallucination rate on MultiHopRAG from 3.29% to 1.02%, while increasing the Arena-Hard-v2.0 creative-writing score from 16.95% to 27.54%.