Quantization method preserves VAE-based policy control and speeds execution
LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantization
Artificial Intelligence
Summary
Video models that help control robots rely on encoded information called latents, which must stay very accurate for the robot's control to work well. The authors found that typical ways of compressing these latents reduce reconstruction quality or cause the robot control to fail. They discovered the main problem is shifts in the latent signals during compression. To fix this, they made a two-step training process called LatentQuant that keeps the latent signals steady and maintains control quality. This method speeds up computation on certain GPUs while keeping robot tasks successful.
What this means in practice
- •For robotics engineers: Use LatentQuant to maintain robot control accuracy when compressing video model latents, enabling efficient and reliable policy execution.
- •For gpu system developers: Implement NVFP4 quantization with LatentQuant to accelerate VAE-based models on NVIDIA B300 GPUs with minimal loss in performance.
Authors
Ziye Deng, Lufang Chen, Shuyu Feng, Zhenwei Duan, Zicong Ye, Yu Sun, Xiaofan Li, Ruyi Gan, Hao Wang, Hao Zhang
Abstract
Recent world action models (WAMs) reuse pretrained video VAEs whose encoder latents directly condition downstream action policies. Quantization must therefore preserve not only reconstruction fidelity but also the policy-facing latent contract expected by the frozen policy. Direct NVFP4 leaves W4A4 quantization error uncompensated, whereas joint quantization-aware training (QAT) can recover reconstruction by moving this representation. On Wan2.1, joint QAT nearly matches FP32 VBench-7 (0.7403 versus 0.7409), yet LIBERO success collapses from 95.5% to 10.5%. Controlled decoder-only experiments show that activation quantize-dequantize operations alter the reconstruction signal and decoder Jacobian, redirecting the gradient returned to the encoder and inducing persistent latent drift. Based on this mechanism, we introduce LatentQuant, a two-stage NVFP4 QAT framework that first aligns the quantized encoder with its high-precision counterpart, then freezes it while adapting the decoder. Across Wan2.1 and Wan2.2, LatentQuant preserves near-baseline control and high reconstruction quality, achieving 95.75% success on LIBERO and 68.8% on RoboTwin. On NVIDIA B300 GPUs, NVFP4 execution achieves 1.17x-1.26x end-to-end VAE speedups over BF16 cuDNN.