LLaCO improves large language models with smarter text generation timing
LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling
Artificial Intelligence
Summary
Generating text with large language models can be tricky because the way words change over time doesn't always match how the model thinks about them. The authors found that measuring how words compete based on direction, not just distance, helps understand when one word should replace another during text generation. They created a method called Semantic Retention Scheduling, which times this replacement better along the generation path. Their new system, LLaCO, uses these ideas to produce more accurate text from continuous representations and performs well compared to existing methods.
What this means in practice
- •For natural language processing developers: Incorporate LLaCO to improve text generation quality and speed in language models using continuous embeddings.
- •For ai-powered customer support teams: Enhance chatbots with LLaCO's more accurate text generation, leading to better understanding and responses.
Authors
Zijian Jia, Peixuan Zhang, Genying Li, Di Wu, Yujie Fu, Si Li, Boxin Shi, Shuchen Weng
Abstract
Continuous diffusion models have emerged as a promising non-autoregressive paradigm for text generation over continuous embedding spaces. Along the continuous trajectory, the residual token identity directly shapes the denoising objective. However, the geometric trajectory misaligns with temporal progression, triggering severe temporal compression. In this paper, we show that embedding norm decay causes Euclidean distance to yield spurious overtaking, indicating that token competition must be evaluated directionally. In this space, a candidate overtakes the source token only when the effective noise magnitude surpasses their semantic separation. We formalize these pairwise dynamics as the Competition Threshold Theory (CTT). Crucially, high-dimensional overtaking requires extreme noise levels, causing standard trajectories to concentrate identity decay into a narrow window near the noisy endpoint. To resolve this, we propose Semantic Retention Scheduling (SRS) to align time allocation with the progression of identity decay, and demonstrate that existing token identity metrics are fundamentally statistical aggregations of these pairwise crossings. Building on these theoretical insights, we introduce LLaCO, a generative framework powered by competition-aware semantic scheduling. LLaCO adapts a pretrained backbone to continuous representations via an embedding directional adapter and timestep-conditioned modulation. To operate within the directional space, LLaCO employs a spherical noising process formulated as a geodesic interpolation toward a mask-biased endpoint, eliminating norm decay while preserving accurate token competition. Guided by SRS, LLaCO reallocates the denoising trajectory proportionally to the logarithmic rank decay. Instantiated as an 8B model, LLaCO outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.