Correctness-gated distillation changes decisions with no clear label benefits
Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation
Artificial Intelligence
Summary
Teaching a smaller AI model using advice from several bigger models can focus on how often their answers are correct and how well their explanations match evidence. The paper looks at a method that weighs teacher advice by correctness but finds it doesn’t guarantee better decisions or explanations in practice. Sometimes, the method lost the ability to recognize certain labels and didn’t improve understanding of the evidence behind answers. The authors conclude that the approach did not show clear benefits over simply filtering out bad advice.
What this means in practice
- •For machine learning engineers: Evaluate multi-teacher distillation methods carefully, as correctness-weighted approaches may alter decision distributions without clear label performance gains.
- •For natural language processing developers: Avoid assuming correctness-based weighting of teacher outputs improves grounding or label recall when building answer generation systems.
Authors
Xiaofei Feng
Abstract
Candidate decision correctness and rationale grounding are different objectives. We examine correctness-gated multi-teacher distillation in a fixed experiment. Eight arms share 4,330 sources, a 63.9M-parameter student, 12,990 optimization rows, 406 updates, evidence inputs, and a decoder; seven teacher-based arms use one fixed three-response pool. Three seeds are evaluated on 267 held-out examples. Relative to unfiltered distillation, the correctness-weighted arm differed in accuracy by +0.1660 (95% observed-matrix interval [0.0670, 0.2455]), five-label macro-F1 by +0.1323 ([0.0916, 0.1731]), and task-defined conditional unsafe-action rate by -0.4979 ([-0.5926, -0.3686]). These shifts do not imply uniformly better behavior. Source-label SFT had the highest mean macro-F1 (0.586). The weighted arm had zero Refuted recall in every seed, and two seeds assigned NotEnoughInfo to all 167 claim examples. In an availability-amended audit at one reference seed, weighted and unfiltered outputs had 0/20 versus 1/20 evidence-supported positives and 20/20 versus 19/20 positives containing unsupported material. Samples were non-paired, source overlap was not serialized, and the amendment followed automatic summarization but preceded annotation. The audit therefore cannot estimate a common-source grounding effect and is inconclusive about system-level improvement or harm. Hard filtering already achieved 0.660 accuracy, 0.530 macro-F1, and 0.135 conditional unsafe rate. The implemented weighted arm showed no demonstrated incremental decision benefit over hard filtering. This fixed-matrix failure analysis shows decision redistribution with lost label functionality; the available human audit does not establish a grounding gain.