Object centered tactile learning improves robot manipulation success
Learning with Object-centric Representations of Tactile Interactive Perception for Robot Manipulation
Robotics
Summary
Robots often have trouble understanding qualities about objects—like softness or if a container is full—just by looking. This paper shows how robots can learn to recognize these hidden object traits by touching and exploring them over time. The authors developed a way for robots to pick out important touch signals and link them with descriptions so the robot gets a better sense of what it’s holding. This learning helps the robot make smarter choices when moving or using objects, even if it hasn’t seen them before. In tests, the robot greatly improved its ability to select the right object and handle it properly.
What this means in practice
- •For robotics engineers: Build robots that better identify and manipulate objects with hidden properties using touch-based learning integrated into control policies.
- •For automation system designers: Design automated systems that adapt manipulation strategies for objects whose properties are not visually obvious by incorporating tactile-driven object contexts.
Authors
Xinyi Yang, Zilin Si, Zhuowei Xu, Zeynep Temel, Oliver Kroemer
Abstract
Implicit object properties that are difficult to directly infer from vision, such as material, container contents, or softness, can be revealed through tactile sensing and exploratory interactions. However, because tactile signals are transient and sparse, extracting informative tactile events and effectively incorporating them into robotic manipulation remains a challenge. In this work, we present an object-centric context-aware manipulation framework that learns task-agnostic object representations through tactile exploration. A token learner autonomously selects representative tactile segments from long-horizon exploration, while contrastive alignment with descriptive text embeddings enables a latent space that captures multiple physical object properties. These learned representations are then used as semantic context to guide object-centric manipulation policies and adapt strategies based on object properties. Experiments show that the learned representations achieve 93% and 84% property estimation accuracy on seen and unseen objects. Evaluated on three tasks involving visually ambiguous objects, i.e. multi-object rearrangement, pouring, and box opening, the proposed framework improves both target selection and property-dependent manipulation adaptation, raising task success, aggregated over all evaluation trials, from 41% to 92% on seen objects and from 19% to 67% on unseen objects over baseline policies without object-context conditioning. Videos and additional results are available at https://xinyiyxyx.github.io/tactile-object-centric/.