Unified vision language models improve text image and image text generation
Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation
Artificial Intelligence
Summary
Unified vision-language models can both create pictures from words and describe images in words using one shared system. The authors studied how different parts of this system focus on the input in each case. They found certain parts, called attention heads, are more aware of the input change, and others are not. Using this, they invented a method to tune these parts during generation, improving results without retraining the model.
What this means in practice
- •For graphic designers: Generate higher-quality images matching complex text prompts by using attention-head level tuning during text-to-image generation.
- •For software developers: Enhance multimodal applications by improving generation fidelity in both text-to-image and image-to-text tasks without retraining.
Authors
Zhengyang Luo, Xiaoyu Qiu, Zhiheng Tan, Yuchen Liang, Yingbin Liang, Liyue Shen
Abstract
Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood. We study this question from the perspective of conditional guidance: we probe every attention head to show the activation difference between two branches of classifier-free guidance (CFG) with different conditional guidance. On two unified VLMs of autoregressive and masked-diffusion paradigms, the probing reveals a subset of heads that are more sensitive to the change of condition while others are agnostic to it. This characteristic pattern holds in both T2I and I2T generation and is largely stable across data samples, demonstrating the internal mechanism of unified VLMs. Motivated by this, we propose a new inference-time intervention approach for unified multimodal generation, named \textit{HeadTuned-CFG}, which uses the probed condition-change awareness to select and tune the conditional guidance at the attention-head level. Experiments show consistent benefits in both multimodal generation directions of unified VLMs without additional training, and in particular substantial improvements over strong baselines in T2I generation. These results further support the key finding of our probing analysis: the observed pattern of functional specialization is an intrinsic property of unified VLMs.