Skill decoupling improves visual language model training for better multitasking
Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models try to understand both images and words, but training them to do many things at once can cause problems when different skills interfere. The authors propose a new method called SCOPED, which trains separate expert models for each skill and then teaches a combined model using these experts without directly showing it the separate skills. This method helps the combined model learn better, avoid confusion between skills, and perform well across many benchmarks. It also helps the model focus more on important visual details and keeps expert knowledge intact.
What this means in practice
- •For ai engineers: Improve training of multitask vision-language models by reducing conflicts between specialized skills for better overall performance.
- •For multimodal application developers: Build more reliable vision-language systems that maintain specialized capabilities without explicit skill signals at runtime across diverse tasks.
Authors
Yuqi Xiong, Chunyi Peng, Zhipeng Xu, Qing Yang, Zhenghao Liu, Zulong Chen, Yu Gu, Ge Yu
Abstract
Reinforcement learning (RL) can elicit specialized visual capabilities in vision-language models (VLMs), yet jointly optimizing heterogeneous objectives often introduces cross-capability gradient conflicts. Training experts independently mitigates this interference, but direct reward optimization may entangle specialized knowledge with reward-hacking biases. This paper proposes Skill-Decoupled On-Policy Expert Distillation (SCOPED), which trains capability-specific experts and extracts explicit skills through reward-guided reflection. During on-policy distillation, these skills guide expert supervision while remaining unavailable to the unified student, encouraging it to internalize procedural knowledge without explicit skills at inference time. Across eight multimodal benchmarks, SCOPED consistently outperforms the backbone and mixed-capability RL, demonstrating the effectiveness of skill-decoupled on-policy expert distillation for integrating independently acquired capabilities. Further analyses reveal interference during joint capability optimization and suggest that skill augmentation promotes more consistent expert trajectories, greater attention to task-relevant visual evidence, and better preservation of specialized expertise in the unified student. All code and data are available at https://github.com/NEUIR/SCOPED.