Robot control adapts safely using combined force and vision sensing
Multimodal Interaction Estimation for Contact-Aware Adaptive Impedance Control
Robotics
Summary
Robots working near people need to operate safely and precisely, even when handling unknown weights or being pushed by humans. The authors created a system that uses both force sensors and 3D vision together to tell whether the robot is holding a load or being pushed by a person. Their method works well even if one type of sensor is blocked or missing. This system makes robots adjust how stiff or flexible they are to respond properly and safely in different situations.
What this means in practice
- •For industrial robot programmers: Enable robots to safely handle unknown payloads and human interactions on manufacturing floors using integrated force and vision-based control.
- •For robotics integrators: Deploy multi-arm robotic setups that adapt stiffness and respond to human intent without retraining for new objects or tasks.
Authors
Mateusz Jaszczuk, Paris Perdikaris, Nadia Figueroa
Abstract
With advancements in learning-based policies, robots are increasingly being deployed in human-centric environments, such as manufacturing floors and homes, where safe and precise operation requires compensating for unknown loads while remaining compliant during human interactions. Reliably distinguishing between these interaction types is challenging with a single sensing modality, as force signals cannot separate payloads from human pushes, and vision is susceptible to occlusions. We propose a task-agnostic interaction estimation and adaptive control framework that addresses this challenge by fusing force history and 3D point cloud through a transformer-based classifier. Trained with modality dropout for occlusion resilience and fine-tuned from simulation to hardware, the classifier achieves 91.89% accuracy and maintains above 90% under complete occlusion of one modality. The classifier drives a system identification framework that selectively compensates for unknown payloads and a variable-impedance controller that switches between adaptive and reactive modes, while simultaneously estimating interaction forces for downstream tasks, such as human intent estimation. We validate the framework under external perturbations and unknown loads across single-arm manipulation, deployment with an imitation-learning-based policy, intent-driven target switching, and zero-shot bimanual deployment on two Franka arms lifting a box of unknown mass, demonstrating generalization to closed kinematic chains and unseen setups, enabling safe and precise deployment.