Optical circuits improve large language model response speed and energy use

Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

Distributed, Parallel, and Cluster ComputingArtificial IntelligenceHardware ArchitectureNetworking and Internet Architecture

Summary

Large language model (LLM) decoding needs fast communication between many processors, but current networks slow things down because they were made for training, not for this decoding. The authors found that the communication pattern during decoding stays the same for a given model setup, so they designed a special optical circuit network that is fixed for each deployment to speed things up. This approach uses fewer communication steps and higher bandwidth by cleverly connecting processors with light-based circuits instead of traditional packet switches. Their tests show this design can make LLM responses up to 65% faster and save energy by more than double in large deployments.

What this means in practice

  • For data center network engineers: Build specialized optical networks configured per workload to boost interaction speed and cut energy for large model inference clusters.
  • For ai service operators: Deploy fixed optical circuits tailored to a given LLM deployment to reduce user wait times and power costs during inference.

Authors

Maximilian Thürmer, Nicolas Scotto di Perto, Eloise Stein, Natanijel Vasic, Mikaeil Bagherli, Zhenguo Wu, Benjamin Klenk, Holger Fröning, Johannes Feldmann

Abstract

LLM inference decode executes fine-grained collectives whose latency directly limits per-user interactivity, yet scale-up fabrics are still built from electronic packet switches designed for training. We observe that, for a given model deployment and parallelization, the communication structure of decode is fixed, and propose workload-static optical circuit switching (OCS): circuits are configured once per deployment, co-designed with model parallelism and with topology-aware collective schedules that turn spare accelerator ports into fewer communication phases or higher bandwidth. We evaluate this design with GXNET, a graph-based workload emulator coupled to analytical and flit-level network models grounded in hardware measurements. In a 576-accelerator scale-up domain serving 5T-parameter MoE models with up to 5M tokens of context, all-OCS fabrics improve interactivity by up to 65% and energy efficiency by up to 119% over 4:1 oversubscribed packet-switched fabrics.