Flow3D-OPD improves 3D shape generation using multi-teacher reinforcement learning
Flow3D-OPD: Multi-Teacher On-Policy Distillation for 3D Geometry Generation with Flow-Matching Diffusion Transformer
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Creating high-quality 3D shapes with AI is hard because it’s difficult to know if the shapes are good and how to teach the AI effectively. The authors developed a method called Flow3D-OPD that uses multiple expert AI models (teachers) to guide a student model in learning better 3D shapes. They first improve the basic model and build special teachers to evaluate 3D shapes, then combine these teachers’ knowledge into one student model to reduce conflicts during training. This approach helps generate better 3D shapes without needing complicated changes. Their experiments show this method improves shape quality consistently.
3D geometry generationflow-matching diffusiontransformer modelreinforcement learningon-policy distillationmulti-teacher distillationdirect preference optimizationgradient interferenceagentic verifiermesh quality evaluation
Authors
Zhiwei Ning, Zhen Zhou, Puhua Jiang, Xintong Han, Gengming Zhang, Jie Yang, Zhonglong Zheng, Yuanjie Zheng, Wei Liu, Chunchao Guo
Abstract
Recent image-to-3D generation models built on flow-matching diffusion Transformers (DiT) can produce high-fidelity meshes, yet their post-training strategy remains largely unexplored. There exist several critical bottlenecks in reinforcement learning: the inherent difficulty of defining comprehensive rewards for 3D geometric quality, and the gradient interference that arises when jointly optimizing heterogeneous objectives. Inspired by the practicability of on-policy distillation (OPD) in large language models and image generation, we propose \textbf{Flow3D-OPD}, a two-stage post-training framework that introduces multi-teacher distillation into 3D geometry generation. In the first stage, we utilize the semi-policy to enhance the foundational capability of the pretrained model and then design an agentic verifier for 3D geometric quality evaluation. Based on the verifier, we could cultivate domain-specialized teacher models via direct preference optimization (DPO). In the second stage, we consolidate heterogeneous expertise into a unified student model through on-policy distillation with hard task-routing sampling and gradient accumulation, which could mitigate the gradient interference in joint optimization. Without relying on elaborate modifications, our straightforward yet effective design achieves consistent improvements across all geometric quality dimensions and surpasses all teacher models in the average metric. Extensive experiments demonstrate that our approach provides an effective paradigm for reinforcement learning in 3D generation.