Optical circuits improve large language model response speed and energy use
Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode
Summary
Large language model (LLM) decoding needs fast communication between many processors, but current networks slow things down because they were made for training, not for this decoding. The authors found that the communication pattern during decoding stays the same for a given model setup, so they designed a special optical circuit network that is fixed for each deployment to speed things up. This approach uses fewer communication steps and higher bandwidth by cleverly connecting processors with light-based circuits instead of traditional packet switches. Their tests show this design can make LLM responses up to 65% faster and save energy by more than double in large deployments.
What this means in practice
- •For data center network engineers: Build specialized optical networks configured per workload to boost interaction speed and cut energy for large model inference clusters.
- •For ai service operators: Deploy fixed optical circuits tailored to a given LLM deployment to reduce user wait times and power costs during inference.