Goal images improve robot learning with fast fine tuning
GIFT: Goal-Injected Fine-Tuning for Efficient Manipulation Policy Adaptation
Computer Vision and Pattern RecognitionRobotics
Summary
Teaching robots to follow instructions and act correctly is hard because they need to understand what outcome to aim for. The authors found that showing robots pictures of the goal they should reach helps them perform better. They created a new method called GIFT that adds these goal pictures into existing robot models without needing a lot of extra training time. This method gently introduces goal information so the robot learns smoothly without losing what it already knows. Tests showed that with just a little extra fine-tuning, robots using GIFT got noticeably better at their tasks.
Vision-Language-Action (VLA) modelsgoal image conditioningfine-tuningconvolutional neural networkgenerative modelsrobot manipulationmodel stabilitysemantic image editingSIMPLER benchmarkLIBERO benchmark
Authors
Xiaoyuan Fang, Shuo Feng, Yuxuan Wang, Enhua Cheng, Peng Zhou, Piji Li
Abstract
Compared with relying solely on initial observations and language instructions, predicting goal images with generative models as high-level visual guidance can significantly enhance the robustness of Vision-Language-Action (VLA) models. However, most existing foundation models have not systematically incorporated goal image conditioning due to the high computational training cost. To this end, we propose Goal-Injected Fine-Tuning (GIFT), a lightweight and efficient fine-tuning framework that seamlessly integrates generated goal images into multiple representative pretrained VLA models. Our approach introduces goal image features into observations via a zero-initialized convolution which progressively grows parameters from zero and prevents harmful noise from disrupting the pretrained policy during fine-tuning. As training proceeds, goal information is gradually incorporated, enabling efficient goal understanding without disrupting model stability. We further introduce a refined image editing method to generate semantically and visually consistent goal images from initial observations and task instructions. Experiments show that goal-aware VLA models achieve substantial performance gains across tasks: with only a single epoch of fine-tuning, GIFT outperforms the base model by 6.0% and 13.4% on two SIMPLER settings, and by 4.7% on LIBERO, demonstrating both efficiency and effectiveness.