Latent interface training improves robots generalizing to new visuals
Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models
Robotics
Summary
Robot models often learn to do tasks by linking how things look to actions directly, which can cause errors when the visual environment changes. The authors developed a two-step training method called Latent Interface Training that first teaches a robot how to act based on goals without looking and then carefully adds visual information in a way that keeps only useful spatial details. This approach helps robots better understand what to do even when the scene looks different than before, improving their performance in both simulated and real-world tests under new lighting and camera angles.
What this means in practice
- •For robotics engineers: Enhance robot systems to maintain task performance despite changes in camera setup or lighting by using spatially guided action generation with constrained visual input.
- •For automation integrators: Deploy robots in environments with varying visual conditions, ensuring consistent task success by separating goal recognition from visual distractions.
Authors
Jianman Lin, Shailesh Shailesh, Zhongyi Luo, Jiafei Duan
Abstract
Robot foundation models achieve strong in-distribution performance but often degrade under visual distribution shifts. When learning to generate actions from pretrained visual representations, models may exploit task-irrelevant visual cues that correlate with demonstrated actions within the training distribution. Such vision-action shortcuts can undermine generalization when these correlations change under distribution shifts. Mitigating these shortcuts requires constraining how visual information is used for action generation while preserving task-relevant spatial information. We propose Latent Interface Training (LIT), a framework-agnostic two-stage strategy that first establishes a spatial-goal-conditioned action prior without images, then constrains visual conditioning through a pose-supervised latent interface. Stage 1 trains the action expert to generate action chunks conditioned on language, robot state, and each demonstrated chunk's terminal SE(3) end-effector pose, learning goal-directed action generation independently of visual cues. Stage 2 introduces a latent interface that aggregates visual and semantic representations and serves as the pretrained action expert's only visual conditioning pathway. The interface is supervised to reconstruct the terminal pose previously used to condition Stage 1, encouraging it to retain the goal-relevant spatial information needed for action generation. Across four vision-language-action and world-action architectures (Pi0.5, MolmoAct2, FAST-WAM, and ImageWAM), LIT improves overall LIBERO-Plus success by 3.87-10.70 percentage points while preserving or improving average LIBERO success. Real-world evaluations show 13.30-16.70 percentage-point gains in success aggregated across three tasks under unseen camera configurations, lighting variations, and distractors.