StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionComputation and LanguageMachine Learning
AI summaryⓘ
The authors studied how large vision-language models (LVLMs) are trained using multiple questions about the same image in a row (multi-turn), but are tested on just single questions (single-turn), causing some problems. They found that this mismatch can make models forget or overfit during training. To fix this, the authors created a method called Stochastic Turn Depth (StochasT) that groups questions in varying sizes during training to better prepare models for both single and multi-turn scenarios. They also designed a new way to test model robustness across different contexts. Their experiments show that StochasT helps models perform well in both situations.
Large Vision-Language Models (LVLMs)Visual Instruction Tuning (VIT)Multi-turn trainingSingle-turn evaluationVisual attention decayContextual overfittingStochastic Turn Depth (StochasT)DropoutBalanced Latin SquareContextual dependencies
Authors
Yuan Qing, Chengzhi Mao, Boqing Gong
Abstract
Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT often packs multiple language tasks about the same image for conversational, multi-turn training, whereas existing benchmarks evaluate LVLMs in isolated, single-turn scenarios. The models can suffer from visual attention decay and contextual overfitting during multi-turn training, making it hard for them to realize their full potential in the mismatched test phase. To close the gap, we propose learning with Stochastic Turn Depth (StochasT), which stochastically groups language tasks for the same image into clusters of varying sizes (turn depth) while preserving their organic order. Hence, while StochasT draws on Dropout and stochastic depth for ResNets, it does not actually drop anything to maximize the utility of the training data. Furthermore, we introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs' robustness under varying contextual dependencies. Extensive experiments demonstrate that StochasT effectively grants LVLMs strong, harmonized capabilities for both single-turn and multi-turn use cases.