MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on improving 3D generative modeling by addressing limitations in how shapes are represented and reconstructed. They introduce MSVS-VAE, a method that makes 3D shape representations more detailed and compact by gradually increasing the density of shape information in a smart way. To decode this information efficiently, they replace a global attention step with a local one that better respects 3D structures and combine information at different scales to capture both overall shape and fine details. Their experiments show that MSVS-VAE is faster and more compact than previous methods, while producing higher quality 3D reconstructions.
3D generative modelinglatent diffusionvariational autoencoder (VAE)voxel representationset-based representationhierarchical upsamplingcross-attentionlocal aggregationmulti-scale decodinggeometry-aware convolution
Authors
Dehao Hao, Kaiyi Zhang, Tanghui Jia, Xiangjun Gao, Dongyu Yan, Weikai Chen, Zeyu Hu, Lingting Zhu, Yingda Yin, Runze Zhang, Li Yuan, Xin Wang, Long Quan
Abstract
High-fidelity 3D generative modeling increasingly relies on the latent diffusion paradigm, where the reconstruction quality of the underlying 3D VAE becomes a primary bottleneck. Existing approaches largely follow two paradigms: sparse voxel-based representations achieve strong reconstruction quality but incur significant memory and computational overhead, while set-based representations are compact and continuous yet typically lag in fidelity due to latent sparsity and excessive global smoothness. We propose MSVS-VAE, a hierarchical set-based VAE that closes this fidelity gap without sacrificing compactness. Our key idea is to progressively densify anchored VecSet latents via hierarchical point-shuffle upsampling, increasing spatial capacity for fine-grained geometry modeling. To efficiently decode from the densified hierarchy, we replace global cross-attention with AVS-Conv, a geometry-aware local aggregation operator operating within local neighborhoods rather than the exhaustive latent set. We further introduce multi-scale query decoding to fuse coarse-to-fine latent features, where coarse scales provide stable global context, and fine scales refine localized geometry, reducing artifacts from overly local receptive fields. Extensive experiments on Objaverse, ABO, and in-the-wild benchmarks demonstrate that MSVS-VAE consistently outperforms prior set-based and voxel-based VAEs, delivering approximately 10x faster decoding than prior set-based methods and approximately 10x higher compactness than voxel-based baselines.