Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuning
2026-08-31 • Computation and Language
Computation and LanguageArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors studied a problem where language models sometimes make up false information when they are trained to copy answers that the original model doesn’t fully know. They introduced methods called knowledge-aligned SFT that limit training answers to facts the model already knows. They tested different ways to do this, including checking answers against real evidence and only keeping information the model can confidently recall. Their tests showed these methods reduced false statements while keeping the model’s overall abilities.
Supervised fine-tuningLanguage modelHallucinationKnowledge alignmentParametric knowledgeEvidence verificationRecall consistencyQwen 3 4BOLMo 3 7BFactuality
Authors
Arthur Becker, Jakob Kemmler, David Thulke, Christine Schäfer, Christian Dugast, Hermann Ney
Abstract
Supervised fine-tuning (SFT) trains a base language model to imitate target responses, and these targets may require knowledge the base model has not robustly internalized. We study this as a source of hallucinations and frame a group of mitigation methods as \emph{knowledge-aligned SFT}: constraining SFT training targets to the base model's parametric knowledge. Under a unified setup, we compare existing generation-based and estimation-based knowledge-alignment methods and introduce two new variants: Evidence Rewrite, which verifies base-model generations using external evidence, and Recall Rewrite, which retains claims only when they can be consistently recalled by the base model. Experiments with Qwen 3 4B and OLMo 3 7B show that knowledge-aligned SFT can reduce factual hallucinations on WildHalu and Biography while largely preserving general capabilities. Recall Rewrite yields the strongest factuality gains and improves refusal behavior on UnknownBench. It thereby confirms that SFT targets beyond the base model's knowledge drive hallucination behavior.