Conflict aware adapters defend teacher models from student copying
Conflict-Aware Logit Adapters for Utility-Preserving Anti-Distillation
Artificial Intelligence
Summary
Sometimes when teaching AI models, the student model learns too much from the teacher's detailed reasoning steps, which can be a problem. The paper presents a method called CALA that changes the teacher model slightly during training so the student can't easily copy its answers exactly. This method carefully tweaks the output only where copying is easiest, without hurting the teacher's ability to give good answers. It works efficiently and doesn't slow down the model much when used.
What this means in practice
- •For machine learning engineers: Protect teacher models in knowledge distillation pipelines by reducing student model imitation without sacrificing output quality.
- •For ai service providers: Maintain the natural fluency and performance of deployed AI models while preventing unauthorized replication by downstream models.$Commercial implications: Enables selling AI model APIs that resist cloning, protecting intellectual property for AI service businesses.
Authors
Quoc Phong Dao, Binh Anh Nguyen, Linh Ngo Van, Thanh-Toan Do, Mehrtash Harandi, Trung Le
Abstract
While reasoning traces improve model capability, they also provide supervision signals for downstream distillation. Initial efforts to mitigate this issue rely on decoding-time interventions, but they incur high inference overhead and can harm the teacher model's natural fluency. We propose Conflict-Aware Logit Adapters (CALA), a lightweight training-time defense. CALA attaches a small residual adapter to a frozen teacher model and trains only the adapter. A proxy student identifies positions where imitation is most sensitive, and the adapter perturbs the output distribution at these positions while remaining within high-probability regions. A conflict-aware optimization strategy balances anti-distillation and utility objectives, while regularization preserves the teacher's performance and fluency. At inference time, CALA incurs modest overhead and requires no auxiliary model. Experiments on reasoning benchmarks show that CALA substantially reduces successful student imitation while maintaining near-original teacher performance and generation quality. These results establish CALA as a practical and efficient alternative to decoding-time approaches.