Intel AI PCs run massive sparse models faster with new engine
Cascadia: Resident 975B MoE Inference on Eleven AI PCs
Artificial IntelligenceDistributed, Parallel, and Cluster Computing
Summary
Large AI models with trillions of parameters usually take a lot of computing power to run. The authors show how to run a very big sparse model using eleven standard Intel AI PCs working together. They created software that spreads the work across CPUs and GPUs efficiently, speeding up calculations and handling long input texts well. This lets them generate text faster than before on normal desktop hardware. Their system also checks that the math stays accurate while running the model this way.
What this means in practice
- •For ai infrastructure teams: Run extremely large sparse AI models across multiple standard desktop AI PCs to achieve faster inference without expensive specialized hardware.
- •For software engineers in natural language processing: Deploy large-scale language models locally with efficient token routing and decoding, supporting very long input contexts and fast streaming output on client machines.
Authors
Tate Berenbaum, Matias Parij, Muthaiah Venkatachalam
Abstract
Mixture-of-experts models make nearly trillion-parameter capacity accessible with sparse per-token computation, provided that the serving system can distribute the weights and coordinate their execution. We present Cascadia's resident execution of Inkling, a 975B-total/41B-active-parameter model, on eleven Intel Core Ultra X7 358H AI PCs, each with 64 GB of memory, Arc B390 integrated graphics and gigabit Ethernet. We contribute a custom resident MoE engine that preserves Inkling's routing rules, constructs compressed graphs for OpenVINO's fused iGPU primitives, and coordinates FP16 expert computation with FP32 output restoration. The engine fits six consecutive decoder layers per machine and represents dense feed-forward blocks as all-active expert slices, reducing measured dense-layer call time from approximately 8.1 to 4.5 ms. A streaming pipeline coordinates concurrent generation, while captured-state draft evaluation measures agreement with the deployed numerical path. Paired measurements at fifteen concurrency levels from 1 to 176 streams reach 60.29 aggregate decode tokens/s at 88 streams, with 46.87 tokens/s over the complete serving phases. At fifteen streams, median first-token latency is 6.05 s. Raising the context budget from the 1,024-position default, real prompts of 1k to 64k tokens recover the embedded code in all 19 measured answers, with first-token time growing as $aN+bN^2$ and decode latency growing approximately linearly, both bounded by a single-threaded CPU attention loop rather than by memory, which holds 512k positions per stream. Evaluation on captured fleet states separates the effects of vocabulary selection and weight quantization on draft agreement. Together, these contributions establish an execution and evaluation approach for large sparse models on distributed client systems with shared CPU-GPU memory.