Knowledge distillation improved by scoring answers in trusted directions
Distilling Directional Verification
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
When teaching smaller language models from big ones, the big model may know a fact well only in one direction, like knowing a child's parent but not the parent’s child. The researchers found that using the big model to pick the best answer in the direction it understands avoids confusing the smaller model. This method leads to more accurate smaller models that better reproduce facts during training. The approach was tested on parent-child facts and showed improved accuracy over previous methods.
What this means in practice
- •For language model engineers: Improve training data for smaller models by using teacher scores only in the relation direction the teacher knows best.
- •For knowledge base maintainers: Enhance automated verification systems by scoring candidate facts in reliable directions to yield better evidence labels.
Authors
Jungseob Lee, Sugyeong Eo, Seongtae Hong, Seungyoon Lee, Chanjun Park, Jaehyung Seo, Heuiseok Lim
Abstract
Knowledge distillation aims to transfer the factual knowledge of large language models to smaller models for efficient deployment. Yet a teacher may recall a relation in one direction while failing to generate the answer in the reverse direction. Distillation from its generated answers can therefore propagate this directional limitation to the student. The same teacher can nevertheless recognize such an answer by scoring the relation in the direction it knows. We introduce directional label distillation, in which frozen teachers score candidate answers in that known direction and the best-scoring candidate becomes the student's training target. On facts about parents and their children, known-direction scoring yields more accurate labels than scoring the requested direction, even after tuned corrections for name priors. With prior-corrected scores, the better direction depends on the facts rather than the template, and reverses on mined facts whose notable entity is the parent rather than the child. With the evaluated children's forward facts withheld, students trained on known-direction labels improve open-ended accuracy on their trained queries by 13 to 15 points over students trained on prior-corrected reverse labels. After generated answers are matched to a fixed name list by lexical similarity, students reproduce nearly all selected labels. Their accuracy largely follows label quality. The label advantage holds on unscreened queries and when candidates are retrieved without inserting correct answers. Our findings show that directional verification mitigates the transfer of errors from teacher-generated answers to students by providing more accurate training targets. Code is available at https://github.com/js-lee-AI/directional-verification.