Pre-trained tensor method improves multi-dimensional image recovery quality
Pre-Trained Low-Rank Tensor Decomposition for Multi-Dimensional Image Recovery
Computer Vision and Pattern Recognition
Summary
Recovering detailed multi-dimensional images usually requires learning each image's structure from scratch, which is slow and costly. The authors propose a method that uses knowledge from a large pre-trained vision model to capture common features across images, combined with a smaller part that learns image-specific details. This approach needs fewer resources, fewer learnable parameters, and recovers images more accurately than previous methods. Their technique strikes a new balance between quality, efficiency, and environmental impact.
What this means in practice
- •For medical imaging professionals: Improve the accuracy and efficiency of reconstructing 3D medical images from partial or noisy data using pre-trained tensor decomposition.
- •For remote sensing analysts: Enhance recovery of multi-dimensional satellite images by integrating pre-trained vision features to reduce computational resources required.
Authors
Bing-Zhang Fu, Zhi-Long Han, Ting-Zhu Huang, Xi-Le Zhao, Deyu Meng
Abstract
Recently, tensor decompositions are prevalent for multi-dimensional image representation, which learn the instance-specific structure of each image from scratch. However, tensor decompositions neglect the common structure across different images, leading to limited semantic modeling capability, high computational cost, and a large number of learnable parameters. To address this challenge, we suggest the first pre-trained low-rank tensor decomposition (PLTD) framework, which organically integrates the pre-trained large vision model into the classical tensor decomposition framework. Beyond the shallow and untrained deep tensor decomposition, the suggested PLTD achieves an unprecedented balance among higher recovery fidelity, fewer learnable parameters, and smaller carbon footprint. Specifically, PLTD factorizes the target tensor into a latent tensor and a learnable transform that maps the latent tensor back to the original data domain. The latent tensor consists of two indispensable and complementary terms, i.e., a fixed pre-trained latent tensor and a learnable low-rank latent tensor. The fixed pre-trained latent tensor is distilled from a pre-trained large vision model (i.e., DINOv3) to capture the common structure of the target tensor, while the learnable low-rank latent tensor characterizes the instance-specific structure of the target tensor. To examine the potential of PLTD, we develop the corresponding multi-dimensional image recovery model and theoretically justify the advantages of this framework. Additionally, we discuss the connections between PLTD and classical tensor decomposition frameworks. Extensive experiments on multi-dimensional image recovery demonstrate that PLTD consistently achieves superior performance compared with state-of-the-art methods.