Skill learning improves visual AI agents with faster task success
ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
Computer Vision and Pattern RecognitionComputation and Language
Summary
Teaching AI agents to learn and reuse visual skills helps them solve tasks more efficiently. Instead of just using text instructions, the authors use images and visual representations to guide the AI’s actions. Their system, ViSkill, stores successful sequences as visual skill cards that the AI can look back on and improve over time. This method helps the AI learn faster and perform better on tasks like puzzles and simple games compared to other approaches.
What this means in practice
- •For robotics developers: Enable robots to learn reusable visual skills from experience to improve real-world task performance and training speed.
- •For game ai engineers: Build game agents that use stored visual skill patterns to solve puzzles more efficiently, improving training and gameplay.
Authors
Hongxing Li, Dingming Li, Yixin Li, Yong Du, Wenqi Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
Abstract
Skill-augmented agents improve sample efficiency by distilling successful trajectories into reusable strategies. Yet most existing approaches remain text-centric, linearizing spatial layouts and action-state correspondences into language that loses critical geometric structure. Recent efforts have begun incorporating visual evidence, but construct and update skills separately from policy optimization, leaving their mutual improvement underexplored. We propose ViSkill, a visual-native skill learning framework that encodes successful interactions as composite visual skill cards directly accessible to VLM agents. Retrieved skills guide both inference and reward shaping, while successful trajectories are distilled back into the library, forming a closed feedback loop in which skill accumulation and policy improvement reinforce each other. An optional cold-start mechanism further accelerates early-stage learning. Evaluated on Sokoban, FrozenLake, and PrimitiveSkill, ViSkill achieves an overall success rate of 0.89, rising to 0.91 with cold-start initialization, outperforming all evaluated proprietary and open-source baselines while converging faster than standard PPO. Our code is available at https://github.com/ZJU-REAL/ViSkill.