Detection transformer accuracy improves with refined teacher predictions
Enhanced Knowledge Distillation for Detection Transformer via Teacher Prediction Refinement
Computer Vision and Pattern Recognition
Summary
Detection transformers are powerful tools for finding objects in images but can be slow and hard to use on small devices. The authors noticed that the way these models learn from teacher predictions can be confusing because sometimes early good guesses get worse later, and some wrong guesses become too confident. They created a way to fix these teacher predictions before teaching a smaller model, making the lessons clearer and more reliable. Tests on popular image datasets show this method helps the smaller models learn better without extra costs.
What this means in practice
- •For mobile app developers: Enable mobile apps to run object detection models more efficiently on smartphones by improving the teaching process from large models to smaller ones.
- •For security camera integrators: Create lightweight object detection models for security cameras that maintain accuracy despite limited processing power by refining teacher model supervision.
Authors
Yitong Xing, Yuhao Cheng, Yanping Li, Yichao Yan
Abstract
Detection Transformers (DETRs) achieve strong performance in object detection but remain challenging to deploy on edge devices due to their high computational cost. Existing DETR distillation methods mainly focus on aligning distillation points, while largely overlooking the quality of the teacher's supervision itself. We observe that due to stage-wise non-monotonic prediction behavior in DETRs, well-localized or correctly classified predictions from earlier stages may degrade in later ones, and some negative predictions become increasingly overconfident. As a result, relying solely on the current stage's predictions yields inaccurate and inconsistent supervision. To address this issue, we propose Teacher Prediction Refinement Distillation (TPRD), a plug-and-play module that refines teacher predictions before distillation by exploiting stage-wise prediction information. TPRD improves supervision quality through Positive Prediction Correction (PPC), which corrects degraded positive predictions by restoring more accurate ones from earlier stages, ensuring reliable localization and classification signals, and Negative Prediction Suppression (NPS) suppresses the influence of overconfident negatives, preventing them from providing misleading supervision to the student. To preserve informative dark knowledge, we further introduce Maximum Dark Knowledge Preservation (MDKP), which selectively refines target-class logits while retaining non-target relations. Extensive experiments on MS COCO and PASCAL VOC demonstrate the effectiveness and robustness of the proposed method. Our code is available at https://github.com/xingyitong1/TPRD.