VisualCoding improves accuracy in visual reasoning tasks by learning reusable skill programs
VisualCoding: Learning Executable Skills for Reliable Visual Reasoning
Computer Vision and Pattern Recognition
Summary
Visual reasoning AI often struggles to give reliable answers because executing steps can lead to errors or made-up responses. The authors created VisualCoding, a method that teaches these AI systems to learn reusable visual skill programs and directly use their answers when applicable. This approach tests and refines skill programs during learning and then runs them carefully to avoid mistakes. Their method performed better than strong existing systems across many different visual reasoning challenges.
What this means in practice
- •For computer vision developers: Integrate VisualCoding to improve the reliability and accuracy of visual reasoning features in AI applications.
- •For software engineers building ai assistants: Use VisualCoding’s program acquisition and execution framework to ensure consistent and verifiable reasoning steps in AI assistants dealing with images or videos.
Authors
Liu Jiaqi, Jin Ruinan, Li Xiaoxiao
Abstract
Learning reusable skills can improve visual reasoning agents, yet applying a skill still requires the agent to implement or invoke a procedure and produce the final answer. Errors and hallucinations at these steps can prevent acquired experience from translating into reliable single-run performance. We propose VisualCoding, which combines execution-guided acquisition of reusable visual programs with direct adoption of their answers on applicable inputs. Execution-Guided Program Acquisition composes visual primitives and additional code into family-level solvers, then iteratively tests and revises them on labeled training instances. Scope-Aware Direct Execution runs a retained solver and returns its valid in-scope answer. We evaluate VisualCoding on four models across four benchmarks spanning 69 task families. VisualCoding exceeds the strongest evaluated baseline in mean accuracy by 4.45 percentage points with Gemini-3.1-Pro and by 14.78 points when reusing the frozen Pro-acquired artifacts with Flash-Lite. Ablations support execution-guided acquisition and show that direct execution outperforms textual or callable reuse of the same programs. Code is available at https://github.com/jiaq-liu/VisualCoding.