Hybrid large language models improve speed and efficiency on demand
Request-Adaptive Execution for Efficient Hybrid LLM Inference
Distributed, Parallel, and Cluster Computing
Summary
Large language models use different internal parts to understand and generate text, but most use the same design for all requests, which can be slow or waste resources. The authors created Morph, a system that adjusts how it uses these parts depending on each request to work more efficiently. Morph mixes two techniques within each section of the model and chooses the best way to process each query ahead of time, improving speed and response quality. Their tests show Morph works better than fixed designs, running faster and handling more requests at once.
What this means in practice
- •For cloud ai service providers: Increase throughput and reduce latency when serving language model queries with diverse request patterns using adaptive execution.
- •For ai application developers: Build language model-powered applications that balance speed and response quality dynamically for different user requests.
Authors
Fahao Chen, Xuzhe Yan, Qi Zhang, Peng Li, Dongxiao Yu, Song Guo
Abstract
Hybrid large language models (LLMs) have emerged as a promising way to combine the modeling capability of the Attention mechanism with the efficiency of state space models (SSMs). However, most existing hybrid LLMs still inherit a static design philosophy: once the Attention-to-SSM layout is chosen, every request must follow the same hybrid execution. This one-layout-fits-all design is inefficient for real-world serving workloads, where different requests rely on contextual information in different ways. This paper presents Morph, a serving system that co-designs the model architecture and runtime for request-adaptive hybrid LLM inference. Its model architecture pairs Attention and SSM layers within each block and uses prefill-oriented gating to select a request-specific layout that remains fixed throughout decoding. Its serving runtime preserves batching efficiency under divergent layer choices through selective batching and enables correct prefix reuse through hybrid-aware context caching. Experiments on six datasets show that Morph achieves a better quality-efficiency trade-off than existing static hybrid baselines. System-level evaluation further shows that Morph improves the serving throughput by up to 3.87x over baselines while substantially reducing inference latency.