Compact robot policies perform well with detailed pretrained vision
Compact Robot Policies Need Fine-Grained Visual Representations
RoboticsArtificial IntelligenceMachine Learning
Summary
Teaching robots to manipulate many different objects works better when their eyes (visual systems) are trained and fine-tuned to the task. The authors found that smaller robot programs still perform well if they use detailed visual information that has been learned beforehand and compressed efficiently. Simply making robot policies bigger or relying on language understanding is less important than having a good visual understanding. This means robots can be smarter and simpler if they have the right visual skills built in.
What this means in practice
- •For robotics engineers: Develop smaller robot control programs that maintain high performance through efficient pretrained visual perception.
- •For autonomous system developers: Improve multi-tasking robots in scenarios with ambiguous goals by integrating language-conditioned visual representations.
Authors
Nanhe Chen, Runqiu Yang, Jiawei Tang, Sichao Liu, Yuquan Wang
Abstract
Multi-task manipulation policies differ in architecture, scale, and pretrained priors all at once, so published comparisons cannot attribute performance to any single component. We argue that most of it comes from the visual representation, and that parameter scale and generative priors are largely incidental. To test this, we build CoRP (Compressed Representation Policy), a deliberately compact policy (48.9M parameters, no vision-language model and no video-generative prior) that factorizes into a representation extractor and a flow-matching action generator. It reaches 97.0% on LIBERO and 75.78%/73.36% on RoboTwin 2.0 Clean/Randomized, matching systems 40.9-163.6x larger. Holding the action generator fixed, we then vary one extractor property at a time. Pretrained initialization is decisive: a random ViT-S/14 drops to 78.1% and an ImageNet ResNet-34 to 74.5% on LIBERO. Pretraining alone is not enough, as freezing the encoder costs 19.8 points. Compression matters as much: resampling each view to 48 tokens beats passing all patch tokens (97.0% vs 83.2%), and a variational information bottleneck over those tokens is worse than a hard token budget, cutting LIBERO-Goal from 95.8% to 33.0% by suppressing the instruction-dependent token selection the policy relies on. Language conditioning contributes only where the observation leaves the goal ambiguous (LIBERO-Goal: 9.2% to 95.8%), while on RoboTwin 2.0, where observations are unambiguous, removing it slightly improves success. Therefore, we argue that a compact policy works when its representation is pretrained, task-adapted, and compressed. Project page: https://corp-policy.github.io/