Your Data Manifold is Secretly a Reward Model: Shell-LCC for Text-to-Video Generation
2026-06-29 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors show that instead of using extra, costly tools to improve text-to-video generation, you can use the structure of good training examples themselves as a kind of built-in guide or 'reward.' They develop a method called Shell Local Coordinate Coding that helps keep generated videos detailed and sharp by focusing on the edge or surface of the data shape, avoiding blur and smoothing. Their approach improves video quality, especially in fine details and reducing motion blur, while being computationally cheaper. Overall, this means better videos without the need for expensive extra feedback.
text-to-video generationdiffusion modelsreward modelsSupervised Fine-Tuning (SFT)data manifoldLocal Coordinate Coding (LCC)Shell Local Coordinate Coding (Shell-LCC)high-frequency detailsmotion blurover-smoothing artifacts
Authors
Shihao Zhang, Yuguang Yan, Junzhe Zhang, Wei Zhao, Bohan Wang, Hanwang Zhang
Abstract
Recent text-to-video (T2V) diffusion models rely heavily on auxiliary reward signals (e.g., via reward models or DPO) to align generated content with human aesthetics and improve realism. These signals, however, incur substantial computational overhead, require costly human annotations, and often yield limited improvement in fine-grained local details. In this paper, we argue that your data manifold is secretly a reward model. By explicitly modeling the manifold structure of high-quality Supervised Fine-Tuning (SFT) data and encouraging video latents to lie on this manifold, we derive dense, differentiable, and nearly cost-free reward signals that significantly improve video quality, particularly in mitigating low-level distortions. Our modeling builds upon Local Coordinate Coding (LCC), which captures the `skeleton' of the manifold. However, directly applying LCC suffers from mean regression, pulling latents toward the geometric mean and losing high-frequency details. We therefore extend it to Shell Local Coordinate Coding (Shell-LCC), which models the manifold `surface' as an isotropic shell to align with the true high-density region. Experiments demonstrate that our approach improves realism, enhances high-frequency details, reduces over-smoothing artifacts, and alleviates motion blur.