When Do Task Vectors Interfere? Mapping the Validity Boundaries of Weight-Space Composition
2026-08-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how combining changes made during model fine-tuning affects the model's behavior. They found that adding fine-tuning changes in parameter space doesn't always lead to simple, predictable changes in what the model does. However, in some tasks and settings, these combinations can roughly predict functional outcomes, especially when considering the type of input and prompt format. Their findings hold across different models, tuning methods, and tasks, but aren't universally reliable for performance prediction. This work helps clarify when and how fine-tuning adjustments interact in language models.
task arithmeticfine-tuningparameter spacefunctional geometrylanguage modelsQwen2.5LoRAmodel adaptationcross-architecture evaluationprompt format
Authors
Chencheng Zhu, Xiaoyang Li, Taotao Cai
Abstract
Task arithmetic treats fine-tuning displacements as composable directions in weight space, yet it remains unclear when parameter addition reflects predictable changes in model function. We separate parameter geometry from functional geometry and measure pairwise functional non-additivity over a two-dimensional task-vector surface, using a first-token predictive-distribution interaction ratio conditioned on an input distribution and evaluated with norm-matched controls, three training seeds, and response-only fine-tuning. On Qwen2.5-1.5B, code+safety is more non-additive than the matched code+math control on code and instruction prompts, but not on math prompts. In a prospectively specified six-task expansion, all eight high-versus-low comparisons of unseen task pairs have the predicted sign. The primary ordering further persists under full-parameter fine-tuning at 0.5B, Qwen2.5 LoRA scale tests up to 7B, and a Llama-3.1-8B cross-architecture audit. External validation exposes a sharper boundary: raw public code, instruction, and safety prompts preserve the continuous contrast, whereas an instruction-style wrapper collapses it on the identical public-code prompts, and EvalPlus pass@1 interactions do not robustly reproduce it. Weight-space composition therefore supports coarse, input- and format-conditioned functional statements across adaptation methods, scales, and one additional model family, not a universal merging-performance predictor.