Llm inference speed improves with flexible hardware operator sharing
HELIX: HEterogeneous LLM Inference with FleXible and Fine-Grained Partitioning
Software Engineering
Summary
Running large language models (LLMs) on computers often involves different hardware parts working together, but traditionally these parts stick to fixed roles which can slow things down. The authors introduce HELIX, a software system that figures out the best way to split up the work across different hardware pieces during runtime, adapting to the model and workload. HELIX breaks down the tasks into smaller pieces to send to the right accelerator for faster results. This flexible approach lets newer, specialized hardware join in without being locked into preset roles, speeding up LLM response times significantly.
What this means in practice
- •For ai infrastructure teams: Reduce latency by dynamically dividing LLM inference tasks across varied accelerators according to runtime conditions and hardware capabilities.
- •For hardware accelerator developers: Integrate emerging accelerators as flexible subgraph executors for LLMs without requiring fixed partitioning roles in existing systems.
- •For digital content platforms: Improve large language model response times for interactive applications by deploying flexible hardware task distribution enabled by HELIX.$Commercial implications: Allows platforms to offer faster AI-driven interactions by optimizing hardware use, creating a competitive product advantage.
Authors
Victoria Clerico, William Simon, Corey Liam Lammie, Orhun Görkem, Athanasios Vasilopoulos, Hsinyu Tsai, Jeronimo Castrillon, Abu Sebastian, Hadjer Benmeziane
Abstract
LLM inference is inherently heterogeneous, with performance bottlenecks shifting throughout inference phases down to individual operators. This has motivated heterogeneous serving systems that disaggregate execution across specialized accelerators, typically through fixed partitions such as prefill/decode or attention/FFN. However, these approaches commit to a partitioning strategy in advance, limiting their ability to adapt to different workloads and model architectures while forcing emerging accelerators into predefined roles. We present HELIX, a vLLM-based software stack that discovers, compiles and dispatches workload-aware partitions across heterogeneous accelerators at the operator granularity. At engine initialization, HELIX automatically determines hardware-specific operator placements that minimize execution latency, accounting for profiled execution costs, communication overheads, device capabilities, and memory constraints. It combines dynamic programming, piecewise compilation, and heterogeneous subgraph dispatch, enabling accelerators with partial operator coverage to participate in LLM inference without predefined model partitions. HELIX provides an open-source unified framework for exploring, evaluating, and integrating heterogeneous accelerators for LLM inference, lowering the barrier to emerging hardware by exposing them as subgraph executors. In systems combining H100 and B200 GPUs with emerging accelerators, HELIX resolves LLM partitions that multi-query decoding latency by up to 3$\times$ for dense and sparse transformer models and up to 3.5$\times$ for hybrid state-space models, relative to standard attention-FFN disaggregation.