DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionMachine Learning
AI summaryⓘ
The authors study how diffusion vision-language models generate responses by gradually removing noise while using visual information at each step, which can be slow and costly. They find that the need for visual evidence changes during the generation process and propose a method called DAVET to smartly adjust how much visual evidence is used at each step. DAVET saves time without much loss in quality by planning and allocating visual information adaptively. They tested this on two models and demonstrated faster performance with minimal decrease in accuracy.
diffusion modelsvision-language modelsdenoisingvisual conditioninginference accelerationadaptive allocationautogressive decodingtrajectory allocationLLaDA-VLaViDa
Authors
Yongkang Zhou, Xiang Xia, Cheng Yan, Fan Xu, Wuyang Zhang
Abstract
Diffusion vision-language models (dVLMs) iteratively denoise masked responses while conditioning each denoising step on visual evidence, making visual conditioning a substantial recurring inference cost. Unlike autoregressive decoding, diffusion generation repeatedly revisits the entire response as uncertainty evolves. Our analysis reveals that visual evidence demand is strongly step-dependent, motivating adaptive allocation across denoising steps. Existing inference acceleration methods operate through decoding-side strategies or visual token compression via pruning and merging, but do not explicitly treat visual evidence as a resource whose demand evolves across the diffusion process. Therefore, we present Denoising-Aware Visual Evidence Trajectory Allocation (DAVET), a training-free framework that allocates visual evidence according to the evolving generation state. Starting from a phase-conditioned evidence trajectory, the proposed allocation policy uses operation demand to set an evidence reserve whose allocation at each denoising step is modulated by trajectory risk. DAVET realizes the resulting budgets through a hierarchy of evidence views constructed from a single visual encoding, separating when and how much evidence is needed from how the evidence views are constructed. Evaluated on two representative dVLMs, LLaDA-V and LaViDa, across multiple visual-understanding benchmarks, DAVET achieves an average speedup of 1.55$\times$ with an average relative performance drop of 1.86\%, showing that denoising-aware visual evidence allocation can reduce visual conditioning cost while largely preserving generation quality.