Fixed-budget probes fail to find practical expert groupings in multi-source training

From Transfer Probes to Expert Grouping: Evidence from LoRA Post-Training

Machine Learning

Summary

When training one AI model for many tasks, it helps to group tasks that can share experts to save resources. The authors tested if training on one task and checking its effect on others (called transfer probes) can reveal good groupings automatically. They found that using these probes to assign tasks to shared experts did not perform better than simply spreading tasks evenly, and both were worse than known groupings. This suggests that in practical settings with limited training time, probe-based grouping may not offer improvement.

What this means in practice

  • For machine learning engineers: Avoid relying solely on initial transfer probes to group tasks for shared expert models under strict training budgets.
  • For cloud platform teams: Design routing systems that do not depend on inferred groupings from probe measurements when training experts for many customer lines.

Authors

Xueyang Wu

Abstract

A platform that post-trains one base model for many business lines often has more lines than experts it can maintain, so lines must share experts, yet lines seldom come with a taxonomy saying which should share. We ask whether transfer probes (train on one source, measure paired changes on all sources) can discover a grouping when the standard one is unknown. To test probes against a reference, we use seven public sources that do have a label tree (five mathematics subfields and two kinds of programming, standing in for business lines) and treat the tree's grouping as a label reference that the evaluator holds and the method does not. With four LoRA experts under fixed per-source exposure and per-expert budgets, we compare a purely probe-derived grouping and an uninformed uniform default (whose tied routing serves every line from one expert) against this reference. Trained once for one interval from a shared warm-up state and routed by a rule fixed by each allocation, neither reached the prespecified practical gain of 2 percentage points, and both had lower overall utility than the reference ($-2.43$ and $-2.54$ pp; 95\% intervals below zero). In a post hoc comparison, the probe-derived grouping was indistinguishable from the uniform default ($+0.11$ pp, $[-1.25,+1.54]$). In fixed-budget LoRA post-training, a grouping built from initial-state single-source probes thus did not deliver the advantage predicted from the probes once carried to a later training state, a finite budget, and specified routing.