Skill specific fusion improves robotic vision and touch integration
What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory
RoboticsComputer Vision and Pattern Recognition
Summary
Robots use both sight and touch to handle objects, but what each sense helps with changes depending on the task stage. The authors studied how to better combine visual and tactile data based on the specific skill the robot is doing, like reaching or grasping. They created a system that selectively pays attention to different sensor information and remembers key past events to improve skill progress. Tests show it detects problems and completes tasks faster than previous methods, especially when touch or memory of past actions matters most.
What this means in practice
- •For robotics engineers: Build more responsive multi-sensor control systems that adapt vision and touch use based on the manipulation skill executed.
- •For automation system integrators: Develop robotic controllers that reduce delay in detecting slippage or task completion during complex contact tasks.
Authors
Amir-Hossein Shahidzadeh, Seungjae Lee, Eadom Dessalene, Shanthosh Raaj Mohanram Mageswari, Soroush Etemad, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos
Abstract
Robotic manipulation integrates vision, touch, and language, whose importance shifts across stages: vision guides reaching, while touch, through its evolution over time, decides grasping, alignment, and contact. Yet existing multi-modal manipulation policies typically use fixed temporal contexts and fusion strategies, despite shifts in what each modality contributes across different skills. We study how vision and touch should be combined at the level of primitive skills, asking what each skill needs from each sensor, and propose a skill-conditioned representation in which the queried skill conditions fusion over modality-specific short-term observation tokens while attending to a sparse event memory that retains terminal observations from the last $K$ executed skills. Evaluated by skill progress estimation on three contact-rich tasks, it reduces slip-detection delay by 87% against fine-tuned SOTA progress models, twist-completion delay by 67.5% against a vision-only ablation, and progress error on a blind search task by 92% through sparse event memory. Gains concentrate exactly where completion is defined by contact or task history. More broadly, our results suggest that observation formation not only policy architecture is a central challenge in multi-modal representation. Project Website: http://what-to-attend-what-to-keep.github.io/