Goal image fine-tuning improves visual robot task performance
GIFT: Goal-Injected Fine-Tuning for Efficient Manipulation Policy Adaptation
Computer Vision and Pattern RecognitionRobotics
Summary
Robots that understand language and vision can perform tasks better if they know what the goal looks like. The authors developed a way to teach robots to use pictures of their task goals without needing expensive retraining. They create clear goal images by editing initial photos using task instructions, then slowly add this information into the robot’s existing brain without breaking it. Their method helps robots complete tasks more accurately and quickly.
What this means in practice
- •For robotics engineers: Improve robot manipulation task success rates by integrating goal images into pretrained vision-language models with minimal extra training.
- •For manufacturing automation teams: Boost factory robot adaptability to new tasks by fine-tuning visual guidance using goal images without full retraining.
Authors
Xiaoyuan Fang, Shuo Feng, Yuxuan Wang, Enhua Cheng, Peng Zhou, Piji Li
Abstract
Compared with relying solely on initial observations and language instructions, predicting goal images with generative models as high-level visual guidance can significantly enhance the robustness of Vision-Language-Action (VLA) models. However, most existing foundation models have not systematically incorporated goal image conditioning due to the high computational training cost. To this end, we propose Goal-Injected Fine-Tuning (GIFT), a lightweight and efficient fine-tuning framework that seamlessly integrates generated goal images into multiple representative pretrained VLA models. Our approach introduces goal image features into observations via a zero-initialized convolution which progressively grows parameters from zero and prevents harmful noise from disrupting the pretrained policy during fine-tuning. As training proceeds, goal information is gradually incorporated, enabling efficient goal understanding without disrupting model stability. We further introduce a refined image editing method to generate semantically and visually consistent goal images from initial observations and task instructions. Experiments show that goal-aware VLA models achieve substantial performance gains across tasks: with only a single epoch of fine-tuning, GIFT outperforms the base model by 6.0% and 13.4% on two SIMPLER settings, and by 4.7% on LIBERO, demonstrating both efficiency and effectiveness.