Salt++ improves few-step streaming video and audio generation quality

Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Generating audio and video in a few quick steps is hard because the system needs to understand both past and future information differently. The authors present Salt++, a new approach that trains models to better handle these challenges by aligning how the system learns from past context and noisy new data. This makes the generation process more accurate and better aligned across sound and visuals, improving quality significantly. They tested Salt++ on video benchmarks and showed it works better than previous methods.

What this means in practice

  • For video processing engineers: Create high-quality short-step audio-video streams with better alignment and motion fidelity for real-time applications using Salt++ post-training.
  • For multimedia content creators: Improve the visual and audio quality of generated multimodal content with fewer generation steps, enabling smoother editing and preview workflows.

Authors

Xingtong Ge, Yutong Wang, Lunjie Zhu, Haitao Lin, Fangyu Lin, Yushi Huang, Xin Zhang, Yi Zhang, Yu Liu, Jun Zhang

Abstract

Few-step streaming audio--video generation requires both causal modeling and step distillation, yet standard training recipes face two context-related challenges. Teacher forcing pairs clean history with a noisy target, but supervises predictive contextual representations only indirectly through velocity prediction. Meanwhile, directly reusing bidirectional score models in causal Distribution Matching Distillation (DMD) creates a mismatch between generation and scoring contexts. We address these challenges with Salt++, a two-stage post-training framework comprising Causal Self-Flow (CSF) and context-aligned autoregressive DMD. CSF exploits contextual information asymmetry by varying the history while keeping the noisy target fixed: a noise-mixed-history student aligns its intermediate representations with those of a clean-history exponential-moving-average teacher. This self-supervised signal encourages the student to extract semantic information and improves cross-modal alignment. Context-aligned AR DMD shares the causal mask and prefix across generator sampling, fake-score training, and real-score evaluation to match generated and reference distributions under a block-conditional KL objective. With calibrated teacher guidance, it performs clean-prefix few-step distillation and then adapts to generated histories without switching objectives or requiring separate consistency distillation. At 480p, Salt++ improves visual and motion quality by 57% and 45% over OmniForcing on JavisBench under the same 4-step causal setting. A separate scale-wise post-training stage extends Salt++ to 4-step $1664\times960$ generation, outperforming bidirectional LTX-2 on six of seven reported metrics. Project page: https://xingtongge.github.io/Saltpp