GelSlim 5 unifies vision and touch in one fingertip camera
GelSlim 5.0: Unifying Vision and Touch with a Single Fingertip Camera
Robotics
Summary
It can be hard for robots to 'see' and 'feel' an object at the same time, especially when cameras outside the hand get blocked. The authors created GelSlim 5.0, a sensor that fits on a fingertip and uses one camera with special lighting to get both visual and touch information. They also made a new computer model called VisTacFusion that combines these two types of data to understand an object’s position and surface shape better. Using both vision and touch together reduces mistakes compared to using just one.
What this means in practice
- •For robotic manipulation teams: Enable robots to more accurately grasp and manipulate objects by combining visual and tactile sensing from a single fingertip sensor.
- •For prosthetics developers: Develop prosthetic hands that provide integrated touch and visual cues to improve object handling and interaction.
Authors
Xiangdong Wang, I-Hsuan Huang, Xiaohao Xu, Xili Yi, Nima Fazeli, Xiaonan Huang
Abstract
Vision and touch provide complementary cues for object pose and local contact geometry, but external cameras are often occluded at the contact site. We present GelSlim 5.0, a fingertip sensor that acquires visual and tactile observations with one fixed camera and a continuous UV-responsive contact layer. By switching illumination, the camera sequentially acquires through-membrane visual images and contact-sensitive tactile images from a shared viewpoint, avoiding inter-camera extrinsic calibration. To combine these observations, we introduce VisTacFusion, a multimodal transformer with separate prediction paths for planar object pose and contact geometry. Asymmetric bottleneck fusion incorporates visual context while preserving tactile spatial features for depth and surface-normal estimation. Modality dropout enables inference with paired inputs or either modality alone. On real validation data from 12 known objects, paired inputs reduce depth MSE, angular error, and translation $L_1$ by 61.2%, 72.3%, and 61.3%, respectively, relative to tactile-only inference with the same model. Relative to visual-only inference, the corresponding reductions are 93.7%, 6.1%, and 8.6%.