Robot camera and joint timing calibrated precisely from single motion

SyncHeC: Markerless Spatio-Temporal Hand-Eye Calibration via Differentiable Rendering and Continuous-Time Joint Optimization

Robotics

Summary

Automatic alignment between a robot’s camera and its moving arm is tricky because it requires knowing both how the camera is positioned and exactly when images were taken relative to the arm’s movements. The authors present SyncHeC, a method that uses detailed modeling and optimization to solve these problems together using just one short motion of the robot. Their approach improves accuracy in robot vision and can even detect tiny timing delays, which helps make robot tasks more reliable. Experiments showed it works across various robots and cameras, improving success rates in robot tasks by correcting timing and position automatically.

What this means in practice

  • For robotics engineers: Calibrate a robot’s camera and arm alignment with timing offset from a single movement to improve task accuracy.
  • For robot control teams: Automatically compensate camera latency and spatial offset during policy training to increase robot task success rates.

Authors

Zhengdong Hong

Abstract

Markerless hand-eye calibration under continuous motion requires jointly resolving camera geometry and the time offset between images and joint states. We present SyncHeC (Synchronized Hand-Eye Calibration), a dense differentiable-rendering framework that estimates both from a single robot trajectory. Continuous-time forward kinematics makes latency differentiable, metric depth complements mask contours, and an offline planner combines spatial sensitivity with temporal excitation. Alternating updates support Eye-to-Hand, Eye-in-Hand, and multi-camera calibration. On a 7-DoF arm, one $6.5$\,s motion yields $0.21$\,cm translation error against a marker-board reference, $0.34$\,cm touch-test error, and $2.7$\,ms latency error relative to an $87$\,ms reference with $95\%$ uncertainty of $\pm1.5$\,ms. Against CM-Calib on matched recordings, translation and nominal latency errors improve by $3.7\times$ and $2.3\times$. Experiments span seven arms and four depth sensors. Compensating the recovered offset during policy training and deployment increases success by $8.1$ percentage points across six families and six tasks ($1{,}800$ matched pairs; paired $95\%$ confidence interval $[6.3,9.9]$ points).
View PDFOpen arXiv