Style-debiased method improves large language model factual updates
Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data
Computation and Language
Summary
Large language models can learn facts from small data sets but sometimes give wrong answers or vary only in wording. The authors propose a new way to train these models so they focus on actual facts, not just style differences. This method helps the model pick more accurate answers when updated with new information. Their tests show it works better and uses much less training data than some previous ways.
What this means in practice
- •For language model engineers: Improve accuracy of fact updates in large language models using less additional training data with style-debiased preference optimization.
- •For knowledge base maintainers: Ensure updated information in language models reflects correct facts over stylistic variations for time-sensitive queries.
Authors
Takayuki Yamamoto, Daisuke Kawahara
Abstract
Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus. The stored knowledge, however, is not always retrieved correctly. We study the eliciting side rather than the storing side: we use preference optimization, which learns from pairs of a preferred (chosen) and a dispreferred (rejected) response, so that the model elicits its stored knowledge more accurately. One proposed approach takes the model's own erroneous response as rejected and the gold answer as chosen, so as to suppress the error. When the target knowledge is partially known, however, most of these rejected responses are factually correct. Using direct preference optimization (DPO) then pushes down rejected responses that contain correct knowledge and differ from the chosen answer only in style, such as length and wording. We propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole. We first test whether, on top of EntiGraph, a representative storing-side method that runs CPT on text synthesized from the corpus, our method adds accuracy efficiently. On QuALITY, the reading-comprehension QA benchmark on which EntiGraph was evaluated, SD-DPO exceeds a baseline we CPT on EntiGraph's synthetic data from the same base model and evaluate with the same procedure. The training tokens this requires are a few dozen times fewer than the additional CPT needed for the same gain. For knowledge updating, the main goal of this work, we use AToKE, a knowledge-editing benchmark for facts that change over time. There, SD-DPO reaches an overall accuracy of 0.982 and answers with the new or the old fact according to the queried period.