Sequential reward scheduling improves text to 3D model training
OuroReward: Sequential Reward Scheduling for Reinforcement Learning in Text-to-3D Generation
Computer Vision and Pattern Recognition
Summary
Making 3D models from text descriptions requires balancing different qualities like matching the description and having clear textures. The authors noticed that fixing all qualities at once can cause conflicts and poor results. They created OuroReward, a method that plans the order of focusing on each quality based on how they affect each other to reduce interference during learning. They also developed AdaSelect, which picks good example descriptions to train on, helping the learning process stay stable. Their tests show these ideas improve the quality of 3D models created from text.
What this means in practice
- •For 3d content creators: Generate higher quality 3D models from text descriptions by improving training dynamics in reinforcement learning.$Commercial implications: Enables development of superior automated text-to-3D generation products for digital artists and game developers.
- •For ai model trainers: Improve training stability and multi-objective optimization in reinforcement learning for 3D generation models by sequentially scheduling rewards and selecting prompts.
Authors
Bingyang Cui, Yujie Zhang, Yiling Xu, Yunfeng Guan
Abstract
Reinforcement learning (RL) for Text-to-3D (T23D) generation requires optimization across multiple quality dimensions such as semantic alignment and texture clarity. Existing methods typically optimize these dimensions simultaneously through multiple reward aggregation, without explicitly modeling inter-dimension dependencies. This can cause imbalanced optimization and persistent interference among conflicting dimensions. To address this limitation, we propose OuroReward, an interference-aware sequential reward scheduling strategy for T23D RL. OuroReward first estimates pairwise dependencies among dimensions and constructs a cyclic optimization path that minimizes cumulative interference. By incorporating the tail-to-head dependency, the cycle captures global compatibility across the entire schedule. Then, OuroReward converts the cycle into a one-pass sequence, and starts optimization from the dimension with the lowest aggregate interference. Rather than assigning a fixed optimization budget to each dimension-wise reward, training adaptively determines when to advance to the next reward according to the remaining optimization headroom of the current one. We further introduce AdaSelect, an adaptive prompt selection strategy that identifies reliable and informative prompts aligned with the model's current capability. By focusing policy updates on these prompts, AdaSelect effectively improves training stability. Extensive experiments across different T23D models and RL algorithms demonstrate that our framework consistently improves generation quality across multiple dimensions.