Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation
2026-05-27 • Robotics
RoboticsArtificial IntelligenceMachine Learning
AI summaryⓘ
The authors tackle the challenge of using touch data in robot manipulation tasks because real-world data is hard to collect and simulations often simplify touch to less detailed forms. They introduce a new way to represent touch called Center-of-Pressure (CoP), which keeps rich contact details while still working well when moving from simulation to real robots. They also develop a method to calibrate tactile sensors without needing exact force measurements. Their CoP approach helps robots with complex tasks like fitting pegs into holes and balancing balls, working better than previous methods and even estimating physical properties like object weight during control.
contact-rich manipulationsim-to-real transfertactile sensingCenter-of-Pressuresensor calibrationdifferentiable dynamicspeg-in-hole taskball balancingmulti-fingered robotic handreinforcement learning
Authors
Jiahe Pan, Stelian Coros, Jitendra Malik, Toru Lin
Abstract
A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data. Sim-to-real reinforcement learning offers a scalable alternative, but the simulation-reality gap prevents information-dense modalities like touch from being effectively used. Existing sim-to-real methods often mitigate this gap by simplifying tactile data into coarse low-dimensional features -- sacrificing the richness required for complex manipulation. In this work, we introduce Center-of-Pressure (CoP), an effective tactile representation grounded in physical principles that preserves dense contact information while maintaining robustness for sim-to-real transfer. To support this representation, we propose a sensor calibration scheme based on differentiable dynamics, enabling the estimation of taxel orientations without requiring ground-truth force measurements. We evaluate CoP on two blind, challenging contact-rich manipulation tasks: peg-in-hole insertion and ball balancing. Across both tasks, policies conditioned on CoP achieve zero-shot sim-to-real transfer on a multi-fingered hand, and outperform both coarse binary-contact and raw-taxel baselines. Analysis of learned policy states further suggests that CoP-conditioned policies encode task-relevant physical properties, such as object mass, as an emergent byproduct of control.