Language model adapts better by routing domain experts dynamically
Upcycling Language Models for Multiple Specialized Domains
Computation and LanguageArtificial Intelligence
Summary
Large language models sometimes struggle when asked to perform tasks from many specialized areas at once. The researchers propose a way to add parts, called experts, that each focus on a specific domain and a router that learns to pick the right expert for each input. This method lets the model perform well across different domains without confusion. They show that just a small number of examples can teach the router to choose experts accurately, improving overall results while keeping the original abilities of the model.
What this means in practice
- •For software developers: Enable AI systems to switch effectively between specialized tasks by routing inputs to domain-specific model parts.
- •For product managers in ai services: Improve multi-domain chatbot or assistant accuracy by dynamically activating appropriate expertise without retraining the entire model.$Commercial implications: Supports commercial AI products that handle diverse domains with higher accuracy and efficiency.
Authors
Mingzi Cao, Xi Wang, Nikolaos Aletras
Abstract
Effective multi-domain adaptation of large language models requires prioritizing relevant domain knowledge while minimizing interference from irrelevant domains. Existing approaches commonly use Low-Rank Adaptation (LoRA) to train a separate adapter for each domain and combine these adapters within one model. However, they either weight adapters equally regardless of the input domain or dynamically weight adapters while still activating all of them including irrelevant ones. Additionally, while efficient, their low-rank structure limits performance. In contrast, upcycling is a model expansion method that adds experts and a router to a dense Transformer layer offering larger capacity than LoRA approaches. In this paper, we propose Domain-Specialized Upcycling (DS-Upcycling) for multi-domain adaptation. We introduce a new domain-aware supervision that allows the router to effectively learn to distinguish domains by selecting the appropriate experts. During fine-tuning on each new domain, we update only the corresponding expert and router while freezing all other parameters. Experiments show that our method achieves the highest average performance across the adapted domains compared to baselines while retaining general performance. With just 256 examples per domain for router supervision, DS-Upcycling achieves 90-98% routing accuracy after adaptation compared with 43-70% for routers trained without our approach.