Orthrus improves throughput and latency for combined embedding and generation models
Efficient Iterative Retrieval with Heterogeneous Batching
Artificial IntelligenceComputation and Language
Summary
Modern search and information systems often use two types of AI models: ones that convert words into numbers (embedding models) and ones that generate text (generative models). These two types usually run separately, making the whole system slower and leaving powerful GPUs underused. The authors created Orthrus, a new system that runs both models together efficiently by mixing different requests in smart batches. Their tests showed Orthrus can handle up to 4.5 times more work and cut waiting times by over half in some cases.
What this means in practice
- •For cloud service operators: Run mixed embedding and generation AI tasks on shared GPUs more efficiently to lower costs and improve response speed.
- •For search engine developers: Build faster search systems that combine vector-based retrieval with generative answers by processing queries in unified batches.
Authors
Dohyun Park, Hubertus Franke, Daniel G. Waddington, Swaminathan Sundararaman, Yongjoo Park
Abstract
Modern information retrieval increasingly employs both embedding and generative models to handle complex queries. However, current serving systems suffer from low throughput and poor GPU utilization because they execute these models in isolation. Coarse-grained partitioning, such as dedicating GPUs to specific tasks, fails to adapt to dynamic workloads and creates computational "bubbles". To address these, we present Orthrus, a serving system that performs heterogeneous batching within a unified inference loop. The primary challenge lies in unifying embedding and generation workloads with conflicting computational patterns while optimizing batch composition for high performance. Orthrus addresses these challenges through chunked embedding with incremental pooling and by adjusting batch composition in a workload-aware manner. Evaluation on four A100 GPUs shows that, relative to baseline deployments, Orthrus achieves 1.28$\times$--4.52$\times$ higher throughput on controlled workloads and up to 55.8% lower end-to-end p99 latency on an iterative-RAG benchmark. We release our code at https://github.com/illinoisdata/Orthrus .