VisualCoding improves accuracy in visual reasoning tasks by learning reusable skill programs

VisualCoding: Learning Executable Skills for Reliable Visual Reasoning

Computer Vision and Pattern Recognition

Summary

Visual reasoning AI often struggles to give reliable answers because executing steps can lead to errors or made-up responses. The authors created VisualCoding, a method that teaches these AI systems to learn reusable visual skill programs and directly use their answers when applicable. This approach tests and refines skill programs during learning and then runs them carefully to avoid mistakes. Their method performed better than strong existing systems across many different visual reasoning challenges.

What this means in practice

Authors

Liu Jiaqi, Jin Ruinan, Li Xiaoxiao

Abstract

Learning reusable skills can improve visual reasoning agents, yet applying a skill still requires the agent to implement or invoke a procedure and produce the final answer. Errors and hallucinations at these steps can prevent acquired experience from translating into reliable single-run performance. We propose VisualCoding, which combines execution-guided acquisition of reusable visual programs with direct adoption of their answers on applicable inputs. Execution-Guided Program Acquisition composes visual primitives and additional code into family-level solvers, then iteratively tests and revises them on labeled training instances. Scope-Aware Direct Execution runs a retained solver and returns its valid in-scope answer. We evaluate VisualCoding on four models across four benchmarks spanning 69 task families. VisualCoding exceeds the strongest evaluated baseline in mean accuracy by 4.45 percentage points with Gemini-3.1-Pro and by 14.78 points when reusing the frozen Pro-acquired artifacts with Flash-Lite. Ablations support execution-guided acquisition and show that direct execution outperforms textual or callable reuse of the same programs. Code is available at https://github.com/jiaq-liu/VisualCoding.