Dataflow system boosts streaming llm inference efficiency and cuts resource use

Accelerated Dataflow: scaling inference in streaming data analytics with efficiency

Distributed, Parallel, and Cluster Computing

Summary

Streaming data systems help businesses analyze data as it arrives, but they struggle when using powerful AI models like large language models (LLMs) that need GPUs. The authors created a new system on Google Cloud Dataflow that better manages such AI workloads by smartly scaling resources and grouping tasks. This system makes processing these streams faster and uses fewer computing resources, improving overall efficiency without slowing down the results.

What this means in practice

  • For cloud infrastructure teams: Improve streaming pipelines that run AI inference by integrating dynamic scaling and efficient batching tailored for heterogeneous computing resources.
  • For financial data analysts: Accelerate real-time pattern detection in streaming financial data using LLM-based inference while reducing resource costs and latency.

Authors

RuiLong Jiang, Dustin Rhodes, Danny McCormick, Yuta Labur, Reza Rokni

Abstract

Streaming data analytics helps businesses identify opportunities and spot risks in a timely fashion. As large language model (LLM) usage grows, people are exploring incorporating agents or LLMs in streaming data pipelines to help find patterns in datasets without supervision or interpret natural languages for content moderation. However, most streaming systems are currently optimized for data movement and perform poorly on GPU intensive workloads such as performing inference with LLMs. We introduce a system on Google Cloud Dataflow designed to tackle hybrid streaming machine learning workloads with co-designed autoscaling on heterogeneous worker pools and prefix-aware batching. This increases throughput 1.5x to 2x for streaming pipelines using LLMs while maintaining similar latency with up to 75% peak resource savings and 84% lower peak tail latency compared with existing streaming data processing systems.