Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets
2026-08-19 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster ComputingArtificial IntelligenceSoftware Engineering
AI summaryⓘ
The authors show how several everyday AI-powered Intel PCs can work together over a typical network to run very large language models that are too big for one PC alone. They split the model into parts, each part running on a different machine, and pass information between them efficiently using special methods to keep speed high. They also handle multiple users at once by mixing their requests smartly. Their approach allows even very large models, like a 70-billion-parameter one, to run interactively on a small group of PCs, with performance and accuracy close to a single large system.
LLM (Large Language Model)Pipeline ParallelismOpenVINOUnified MemorySpeculative DecodingMicro-batchingGPU OptimizationShardInferenceBeam Search
Authors
Tate Berenbaum, Muthaiah Venkatachalam
Abstract
Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of AIPCs, working together over an ordinary network, can serve models beyond the capability of any single one. We use pipeline parallelism: a model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so that every machine runs one shard and passes activations to the next. Three techniques make this fast enough to be useful. First, we recover the speed of the unsplit model: a naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization, and injecting a beam_idx Gather into each shard triggers that optimization (the IndirectKVCache fusion) and brings the shards to parity. Second, we leverage speculative decoding on stateful OpenVINO models. Third, the pipeline serves several users at once by interleaving their requests across the stages, each request carrying its own cache (micro-batching). Together, a two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user throughput of the unsplit model on the same hardware, and the gap widens under simulated wide-area latency. The same design scales to a 70B model that no single fleet member can hold: a four-node deployment of Lunar Lake AI PCs on Intel Tiber Cloud serves a single user at interactive speed, with output token-for-token identical to the same four-node pipeline decoding without speculation. Code, raw benchmark logs, and reproduction scripts ship as a self-contained package at https://github.com/labscommunity/pipeline-sharded-inference-paper (in the top-level reproduction/ directory).