Seeing Through Extreme Visual Sparsity: Surface Understanding from a Single Random Visual Patch
Computer Vision and Pattern Recognition
Summary
The authors studied how to recognize surface materials and reconstruct images when only a small part (10%) of the original image is visible. They tested four pre-trained models (ConvAE, ViT, Swin Transformer, and MAE) adapted to both tasks of rebuilding the image and classifying materials. Their experiments showed that the Swin Transformer was best at identifying materials, MAE was best at reconstructing images, and ViT balanced both tasks well. All models worked fast enough for real-time use. The authors highlight that while it is possible to recognize materials with very limited visual data, fully accurate image reconstruction remains difficult.
Authors
Sindhuja Penchala, Sudip Mittal, Noorbakhsh Amiri Golilarz
Abstract
Surface material recognition from incomplete visual observations remains a challenging problem in robotic perception and environmental understanding. This paper discusses Sparse Surface Understanding Framework (SSUF), a unified dual-task learning framework that adapts four pretrained architectures-Convolutional Autoencoder (ConvAE), Vision Transformer (ViT), Swin Transformer, and Masked Autoencoder (MAE) for si-multaneous surface reconstruction and material classification. Experiments were conducted on the Touch-and-Go dataset using a sparse observation protocol in which only 10% of the original image remained visible while the remaining regions were masked. To enable a fair comparison, reconstruction-oriented models were extended with classification heads, whereas classification- oriented models were augmented with reconstruction decoders. The resulting architectures were assessed using reconstruction quality, classification performance, model complexity, and in-ference efficiency metrics. Experimental results revealed distinct strengths across the models. Swin Transformer achieved the best classification performance with an accuracy of 89.21%, an F1-score of 0.8922, and a ROC-AUC of 0.9813. In contrast, MAE produced the highest reconstruction scores among evaluated models, with a PSNR of 16.06 dB and an SSIM of 0.4501, while ViT provided the best overall balance between reconstruction and classification performance. Furthermore, all models achieved real-time inference, requiring less than 5 ms per image. Over-all, the results show that pretrained architectures can support material recognition under severe visual sparsity, while accurate image reconstruction remains challenging.