SeMo3D improves 3D asset generation editing and completion tasks
SeMo3D: Unifying 3D Generation, Editing, and Completion via Selective Modeling
Computer Vision and Pattern Recognition
Summary
Creating and modifying 3D models usually means remaking the entire object, even if only a small part needs to change, which can cause problems with consistency. The authors noticed that whether making new parts, editing, or filling in missing sections, the process involves focusing on a selected area while keeping the rest the same. They built SeMo3D, a method that works on any of these tasks by targeting only the regions that need work while preserving the rest. This approach also allows training without specifically paired examples for editing or completion, making it more flexible and efficient.
What this means in practice
- •For 3d content creators: Produce or modify parts of 3D assets more efficiently while keeping other regions unchanged, improving tool consistency and flexibility.
- •For virtual reality developers: Integrate precise 3D object generation and editing guided by text or images to create immersive and controllable environments.
Authors
Xiaokun Sun, Jian Yang, Zhenyu Zhang
Abstract
3D generation, editing, and completion are typically treated as regenerating the complete 3D asset under task-specific constraints, even when only a local region needs modification. This leads to task isolation, inconsistency in content that should remain unchanged, and dependence on task-specific paired data. We address these issues with a key observation: the three tasks all synthesize a designated region while keeping the rest intact---the full region for generation, the target region for editing, and the missing region for completion. We term this paradigm Selective Modeling and build SeMo3D upon it; it naturally unifies the three tasks, preserves unchanged content by construction, and enables training without paired editing or completion data, since any 3D asset can be partitioned into a selected region and its context. For training, we propose region-selective denoising, which randomly selects regions of varying sizes on standard 3D assets and applies noise and loss only to them, jointly learning all three tasks. For architecture, we propose Omni3DiT, a multi-condition 3D diffusion transformer that distinguishes regions to create from those to retain and fuses image, text, and point guidance for fine-grained control. Experiments across three benchmarks show SeMo3D surpasses state-of-the-art methods on all tasks, delivering multimodal controllability in generation and strong consistency and accuracy in editing and completion. SeMo3D further serves as a unified modeling tool for agents, enabling efficient 3D creation through natural-language interaction. Project page: https://xiaokunsun.github.io/SeMo3D.github.io/.