Semantic consistency improves knowledge distillation in driving AI models
RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation
Computer Vision and Pattern Recognition
Summary
Teaching smaller AI models to understand driving instructions can be tricky because copying everything from bigger models may spread mistakes and confuse important details. The authors propose a method called RSKD that picks out key phrases related to driving tasks and checks if the big model’s answers match the truth. If the answers are reliable, it learns from them; if not, it trusts the correct information more. Tests on different driving question datasets show that this approach helps smaller models perform better.
What this means in practice
- •For autonomous vehicle engineers: Enhance compact driving AI by selectively using reliable knowledge from large models to improve understanding of navigation tasks.
- •For robotics software developers: Refine multimodal model training to better capture task-relevant language cues in navigation and decision-making modules for robots.
Authors
Chuchu Xie, Yang Zhong, Kangan Qian, Hao Ye, Kun Jiang, Diange Yang
Abstract
Knowledge distillation can transfer large multimodal models to compact models for autonomous driving, but uniform token-level matching can dilute planning-relevant semantics and amplify teacher errors. We propose RSKD, a semantic-consistency-guided distillation framework that routes supervision according to both the task importance of an answer phrase and its semantic consistency with the teacher prediction. Task-critical semantic phrases are localized in the ground-truth answer, and a phrase-level consistency score controls the balance between Jensen--Shannon divergence (JSD) distillation and ground-truth supervision. Teacher information is emphasized only where it is semantically consistent, while semantically inconsistent phrases receive stronger ground-truth supervision. We evaluate RSKD on DrivingVQA, LingoQA, and MapLMv2. RSKD improves over the compared student-based baselines on DrivingVQA and LingoQA and remains competitive on MapLMv2.