Patch Policy: Efficient Embodied Control via Dense Visual Representations
2026-07-20 • Robotics
RoboticsMachine Learning
AI summaryⓘ
The authors introduce Patch Policy, a new way for robot controllers to use detailed visual information from Vision Transformers without slowing down. Instead of just looking at one big picture summary, their method lets robots consider many small image pieces at once, improving how they understand their surroundings. This approach is faster and smaller than previous big models but works better in many tests both in simulations and real robots. Overall, the authors show a way to use advanced vision skills while keeping robot actions quick and efficient.
Vision Transformersrobot learningdense visual featurespatch tokenstransformer policiesvisual backbonetemporal causalityvision-language modelsrobot control
Authors
Gaoyue Zhou, Zichen Jeff Cui, Ada Langford, Bowen Tan, Yann LeCun, Lerrel Pinto
Abstract
Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io