Multi subject video generation gains precise control and better identity consistency

Harnessing Intrinsic Subject-Aware Attention for Controllable Multi-Subject Video Generation

Computer Vision and Pattern Recognition

Summary

Generating videos with multiple people or subjects is hard because it’s difficult to control how closely the video matches the input references and to avoid mixing up who is who. The authors studied how a type of AI model focuses on different parts of the video and found it naturally highlights each subject’s location. Using this, they created a method that guides the model during generation to keep subjects clear and consistent. Their method also uses rewards during training to prevent the model from drifting away from the subjects. This leads to videos that better maintain each subject’s identity and allow users to control the quality without needing to retrain the model.

What this means in practice

  • For visual effects studios: Create multi-character video scenes where each character’s look and location can be precisely controlled during generation.$Commercial implications: Enables studios to produce higher-quality controlled character animations for films or games, improving workflows with less manual editing.
  • For advertising agencies: Generate marketing videos featuring multiple identifiable products or people with consistent appearance and controlled emphasis without additional model training.

Authors

Niange Yu, Ye Tian, Biaolong Chen, Miao Lu, Aixi Zhang, Hao Jiang, Yunhai Tong, Pipei Huang

Abstract

Multi-subject video generation faces two key challenges: uncontrollable fidelity strength and potential semantic drift. We address these by analyzing the internal mechanisms of Diffusion Transformers (DiTs). We found that certain attention blocks naturally form an Intrinsic Spatial Grounding Map (ISGM) that precisely locates reference subjects. Building on this insight, we propose Dual-phase Intrinsic Attention Leveraging (DIAL), a framework that uses these internal signals for both training and inference. In low-noise stages, we use ISGM to guide the attention mechanism, allowing precise control over fidelity strength during inference without retraining. In high-noise stages, we use these same maps to automatically build preference pairs at no additional cost for Reinforcement Learning (RL). This RL procedure effectively anchors the model's attention to reference subjects and mitigates semantic drift. Extensive experiments show that DIAL significantly outperforms baseline models on the OpenS2V-Eval benchmark, consistently improving identity consistency and enabling controllable fidelity strength.
View PDFOpen arXiv