HydraLLM enables flexible model sizes for text and vision language tasks
HydraLLM: Adaptive Width Slicing for Elastic Text and Vision-Language Models
Computation and LanguageMachine Learning
Summary
Models for understanding language and images often have fixed sizes which make them hard to use on different devices with varying capabilities. The authors propose HydraLLM, a new design that lets the model adjust its size dynamically by turning on or off attention heads, which affects the whole model’s processing layers. This flexible design allows one model to work efficiently on both powerful servers and smaller edge devices while keeping good accuracy. The authors also show it can handle vision and language together with this elastic sizing, improving speed without greatly sacrificing quality.
What this means in practice
- •For edge device developers: Run the same language or vision-language model on devices with very different computing power by adjusting the model size on demand.
- •For data center ai engineers: Deploy a single adaptable model that improves inference speed and resource use by dynamically selecting active model components based on load and latency needs.
Authors
Oscar Kemme, Janek Haberer, Ali Hojjat, Olaf Landsiedel
Abstract
LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fixed-size architectures limit the ability to reuse a single model across deployment settings with different compute, latency, and model-size targets. We introduce HydraLLM, an elastic MQA-based architecture. Its number of active attention heads determines the effective widths of the embeddings, attention projections, feed-forward layers, and the output head, forming a nested family of subnetworks within a single parameter set. We introduce a static-to-elastic conversion pipeline that, starting from a pre-trained model checkpoint, includes importance-driven weight reordering and stochastic subnetwork training with knowledge distillation. We extend this principle to multimodal inference through an elastic projection module whose output width follows the active text width. HydraLLM covers a wider range of active parameter configurations than FFN-only elastic architectures. With 75% of heads active, HydraLLM achieves 2.2 percentage points higher TriviaQA accuracy than a state-of-the-art nested baseline configuration which has 17.25% more parameters. With 50% of heads active, our multimodal model achieves a CLIPScore comparable to that of the 7$\times$ larger Gemma 3 4B model. With only 25% of heads active, HydraLLM increases token-generation throughput by 75.3% over the full-width configuration.