Hybrid network improves fast 3d surface mapping from single image
Hybrid Mamba-Transformer Network for Single-Shot 3D Reconstruction from Fringe Patterns
Computer Vision and Pattern Recognition
Summary
Capturing the exact 3D shape of an object from just one patterned image is hard because details must be clear and consistent over large areas. The authors created a new hybrid network combining special transformers and convolutions to better understand and preserve these details. Their method is more accurate than previous models when tested on real data. This shows a new way to quickly and precisely map 3D surfaces from a single image.
What this means in practice
- •For industrial quality inspectors: Obtain high-accuracy 3D surface measurements from a single fringe image for fast defect detection on production lines.
- •For augmented reality developers: Generate precise 3D models of real-world objects quickly from one image to enhance AR object integration and interaction.
Authors
Xinjie Zhou, XiaoLei Li
Abstract
Deep learning facilitates rapid single shot three dimensional (3D) reconstruction using fringe projection profilometry (FPP), but recovering accurate surface height from a single high frequency fringe image remains challenging because both long-range spatial consistency and fine geometric details must be preserved. Convolutional networks are limited in modeling long range dependencies, whereas window-based Transformers establish global interactions only progressively with increasing depth. To address these limitations, a hybrid Mamba Transformer network is proposed for fringe to height regression. The network adopts a Swin based U shaped architecture and introduces visual state space modeling at the intermediate resolution encoder stage. An axial and dilated vision state space (ADVSS) block is further developed to enhance prescan feature encoding through local, dilated, and axial depthwise convolutions while retaining four directional selective scanning. This design combines linear complexity long range modeling with direction aware preservation of local fringe structures. Experiments on a publicly available FPP dataset show that the proposed method outperforms hNet, SwinConvUNet, UHRNet, and HiDNet in mean absolute error(MAE), root mean square error (RMSE), absolute relative error, and structural similarity index measure (SSIM). Using only one deformed fringe image, the proposed method achieves an RMSE of 0.548 mm and an SSIM of 0.9985, reducing RMSE by 65.2% and 59.6% relative to UHRNet and HiDNet, respectively. These results demonstrate the effectiveness of combining selective state space modeling with hierarchical window based attention for accurate single-shot FPP reconstruction.