RoSetta method aligns features across different sensor types without training
ROSETTA: Rotation Subspace Estimation for Training-free Transmodal Alignment
Computer Vision and Pattern Recognition
Summary
Matching data from different sensors like cameras and LiDAR is hard because each sensor encodes information differently, with no common reference point. The authors discovered that differences between these sensor encodings can be described as a rotation in a transformed feature space that changes unpredictably over time. They developed RoSetta, a method that can estimate these rotations without any additional training or needing simultaneous data from both sensors when querying. This approach works quickly and was tested on multiple types of sensor data across various driving datasets.
What this means in practice
- •For autonomous vehicle engineers: Align sensor feature data from different modalities in vehicle perception systems without retraining encoders.
- •For robotics system integrators: Calibrate multi-sensor platforms quickly by estimating relative rotations between sensor features without coobservations at runtime.
Authors
Alex Junho Lee
Abstract
When a query must be matched against a database built with a different sensor or processed by a different encoder, the independently trained encoders impose distinct coordinate systems on the feature space. No canonical reference frame exists to which all encoders can be aligned, and no single training procedure has fully closed this gap. We show that the relative transformation between any two such systems is a spatially preserving rotation in whitened feature space, whose angular velocity exhibits white-noise characteristics: large in norm, temporally uncorrelated, and unpredictable from the features themselves. However, the cross-covariance between encoder outputs has low effective rank, constraining only a small fraction of the rotation parameters, therefore local alignment is tractable. ROSETTA is a training-free calibration layer that estimates the local rotation via an expectation-maximization procedure, requiring no coobservation at query time. We validate ROSETTA on three public driving datasets spanning LiDAR, camera, thermal, and event modalities in cross-modal, cross-device, cross-representation, and cross-backbone settings, with milliseconds overhead per query.