Task vectors from different learning methods combine well across tasks
No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation
Machine Learning
Summary
Getting robots or AI models to learn new tasks often means blending different skills, but it’s unclear if updates from one training style work well when combined with another. The authors studied how task updates from on-policy distillation (OPD), a way models learn from themselves using feedback, fit together with updates from reinforcement learning (RL). They found that even when OPD models alone perform worse, their task updates can mix with RL updates to create better skill blends. This means weaker models can still add useful, different improvements when combined with stronger ones across multiple tasks.
What this means in practice
- •For machine learning engineers: Combine task updates from different training methods to improve multi-task model performance without retraining from scratch.
- •For ai system integrators: Design systems that merge learned skills efficiently by leveraging complementary updates from OPD and RL-trained modules.
Authors
Jingang Zhou, Feiyu Han, Han Zhu, Yuyi Zhou, Ruiyang Zhang, Jian Xu, Sirui Gao, Qingpei Guo, Xu-Yao Zhang
Abstract
Task vectors provide a simple mechanism for composing learned capabilities through model merging. However, the composability of task vectors produced by on-policy distillation (OPD) remains largely unexplored. OPD trains a student using teacher feedback on student-generated trajectories, yielding parameter updates that differ from those produced by the teacher model, usually by reinforcement learning (RL). We therefore ask whether OPD task vectors can complement their RL teacher updates and compose effectively across tasks. Across five domains and two model architectures, we find evidence for both forms of composability. Within a task, merging OPD and RL task vectors can outperform both constituent models, even when the OPD student is weaker than its RL teacher. Across tasks, OPD task-vector compositions achieve higher average scores than corresponding RL compositions in seven of eight backbone-merging-rule comparisons. Parameter-space analyses reveal substantial non-collinearity between OPD and RL updates. Experiment in CODE domain on SMOLLM3-3B shows that the combined direction outperforms either constituent direction at the tested global update norm, supporting directional complementarity in this configuration. Across tasks, OPD updates also show lower overlap among the top-10% feed-forward channels ranked by update energy. Together, these results show that weaker standalone performance does not imply weaker task-vector composability. OPD task vectors can complement stronger RL teacher updates and combine effectively across tasks, highlighting composability as a distinct property for understanding and evaluating post-training updates.