Robot fingertip sensors improve imitation learning through new alignment method
Self-Supervised Anchoring of Fingertip Sensing to Proprioception and Proactive Actions for Robot Imitation Learning
Robotics
Summary
Robots often learn tasks by watching cameras, but they struggle to sense when their fingertips touch or get close to objects because of blind spots and timing limits. The authors show that adding fingertip sensors that measure pressure and proximity can help, but only if the robot learns to use these sensors in the right task phases. They developed a method called PROPRA that helps the robot connect fingertip sensor signals with its own movements and positions, making the sensors more useful. This method improved robot success in real manipulation tasks by better using fingertip information before and after contact.
What this means in practice
- •For industrial robotics teams: Enhance robot grasping and manipulation accuracy by incorporating fingertip sensing aligned with robot movements.
- •For robotics hardware developers: Design and integrate pressure and proximity fingertip sensors with pretraining techniques to improve robot interaction feedback.
Authors
Tomohiro Motoda, Masaki Murooka, Keisuke Shirai, Hanbit Oh, Ryoichi Nakajo, Shotaro Miwa, Roman Mykhailyshyn, Hugo Duarte, Yukiyasu Domae
Abstract
Robotic imitation learning often relies on external cameras, yet local interaction cues such as object proximity, contact onset, and grasp state are difficult to observe near the fingertips because of occlusion and limited temporal resolution. We study how to effectively incorporate complementary fingertip sensing into imitation learning using pressure-sensitive tactile and reflective proximity sensors, along with pretrained sensor encoders. The two modalities provide information at different manipulation phases: proximity sensing is informative before contact, whereas tactile sensing becomes informative after contact. However, naively adding these signals to a policy does not consistently improve performance and can even underperform vision-only policies, suggesting that sparse, phase-dependent sensor signals are difficult to exploit from limited demonstrations. We therefore propose a proprioception-anchored pretraining method, PROprioceptive-and-PRoactive Anchoring (PROPRA), which independently aligns each fingertip sensor history with proprioceptive and action segments. This provides a continuously available sensorimotor reference, allowing each sensor to be aligned independently during its informative phases. Experiments on real-world manipulation tasks show that our pretraining method improves average success rates over vision-only policies and image-anchored pretraining baselines. Representation analysis further shows that it preserves richer information about pre-contact states, enabling more effective use of complementary fingertip sensing. Please refer to our project page: https://tomohiromotoda.github.io/nia.propra/