Local vision language model controls robot arm and improves with practice
Generalizing Manipulation Skills with a Local Coding Agent
Robotics
Summary
Teaching robots to do new tasks usually requires lots of work or data. This paper shows a method where a local vision-language AI writes and runs its own code to control a robot arm for tasks with toys. The system can handle new tasks it hasn’t seen before and gets faster when repeating tasks. The authors also discuss current limits and ways to improve this approach for real-world use.
What this means in practice
- •For robotics engineers: Control robotic arms for new manipulation tasks without extra programming or training by letting a local AI generate and run its own code.
- •For embedded systems developers: Integrate local vision-language models to create adaptable control software for robots running on a single workstation.
Authors
Raman Talwar, Elias Nijs, Andreas Verleysen, Francis wyffels
Abstract
Today, progress in open-weight language models enables systems capable of writing, executing and debugging code while still running on a single workstation. Most language-driven robots give the model a fixed action interface or a trained policy. Generalizing to a new task therefore means more engineering effort or more data collection, both time-consuming. We investigate whether a local open-weight vision-language model can control a robot and one-shot generalize to new variations of a task without new human programming or training. We let a local open-weight VLM, Qwen3.8-27B, drive a UR3e robotic arm from a coding-agent harness. It writes and runs its own code above a service that implements kinematics, safety limits and classic computer vision techniques. We investigate if this system is capable of generalizing to unseen tasks. Specifically, we test it on nine tasks built from children's toys designed to probe generalization capability across various object characteristics: color, size, shape, and task variation of those. With five trials for each task, we observe generalization in 30 out of 45 trials with durations ranging from 3.4 to 67.5 minutes depending on task complexity. We further test if there is a speedup when an agent is asked to redo the task after successful completion. This resulted in a 50% reduction in duration, indicating that there is self-improvement over time. Finally, we expose the limitations of a local coding agent. We believe that solving those limitations combined with further investigation of self-improvement over time points at a direct path toward real-world deployment of a local coding agent.