Language models manage their own tasks to speed up inference
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Artificial IntelligenceComputation and Language
Summary
Large language models are powerful but slow because they generate one word at a time, which makes them hard to use efficiently. The authors propose models that mark which words depend on others while they generate text, letting the computer run some parts in parallel or free up memory earlier. They build three systems showing how this idea can make language models faster or use less memory without losing quality. This way, the models decide for themselves how to balance speed, memory use, and accuracy during generation.
What this means in practice
- •For ai infrastructure engineers: Deploy language models that dynamically optimize generation speed and memory use by exploiting token dependence annotations to improve hardware efficiency.
- •For cloud service operators: Run scalable language model services with reduced latency and memory footprint by leveraging models that self-organize their inference tasks.
Authors
Tian Jin
Abstract
Large language models (LLMs) demonstrate impressive capabilities, but their deployment presents significant efficiency challenges. Autoregressive decoding imposes substantial inference latency and under-utilizes hardware accelerators in low batch size regimes. Discrete diffusion models can generate in parallel but struggle to match autoregressive quality without many diffusion denoising steps. Long-context reasoning creates memory bottlenecks that strain even state-of-the-art accelerators. My thesis is that language models can direct their own inference execution strategy by annotating semantic dependence -- which tokens depend on which others -- in their generation. I call such models self-orchestrating language models. For each system, I design a runtime that acts on these annotations to parallelize autoregressive decoding, evict intermediate context, or derive denoising orders, achieving Pareto-optimal quality-efficiency trade-offs. I demonstrate this approach through three self-orchestrating systems. First, PASTA uses semantic dependence to parallelize autoregressive decoding, training the model to annotate which output chunks can generate independently. Second, TIP uses semantic dependence to evict intermediate reasoning steps from the KV cache, reducing memory consumption while preserving accuracy. Third, Planned Diffusion uses semantic dependence to derive a denoising order for discrete diffusion, autoregressively generating a plan that specifies which chunks to denoise in parallel.