Person actions help predict group intent in vision language models
Do the Details Help the Whole? One-Way Transfer from Person Actions to Group Intent in VLMs on JRDB-Social
Computer Vision and Pattern RecognitionComputation and Language
Summary
This paper looks at whether knowing what each person in a group is doing helps a computer model understand what the group as a whole intends to do. The authors focus on vision-language models using a dataset called JRDB-Social. They find that providing details about individual actions can slightly improve the model's ability to guess the group's intent, but knowing the group's intent does not help recognize the actions. Correct and specific details about people’s actions are important for this improvement.
What this means in practice
- •For video surveillance teams: Improve group activity recognition by incorporating recognized individual actions to better predict collective intent in monitored scenes.
- •For social robotics developers: Enhance robots’ understanding of group behaviors by enabling models to infer group intent from observed individual actions.
Tested on one dataset.
Authors
Chengguang Gan, Zhixi Cai, Yunhao Liang, Hanjun Wei, QingHao Zhang, Shiwen Ni
Abstract
Social scene datasets label people at two levels: what each person is doing and what the group is there for. Training on both levels at once is often assumed to help each of them. We test this assumption for vision-language models (VLMs) on JRDB-Social, treating person actions as the details and group intent as the whole. We fine-tune Qwen3.5 4B under eight training arms with three seeds, repeat the core arms on Qwen3.5 9B and Qwen3-VL 4B, and judge every arm with a verdict rule fixed before testing. Writing each member's actions before the intent raises intent macro-F1 from 30.1 to 31.0, and writing the intent first reaches 31.2, while action F1 stays within 0.2 points of an action-only model. Under the selected recipe, no arm trained with intent improves actions. The gain requires correct, image-grounded actions: empty action lists or actions borrowed from another group with the same intent lower intent F1. Forced-prefix tests, occlusion, member masking, linear probes and component grafting show that the intent follows the written actions, relies more on people and partly resides in the vision tower. Supplying actions as input instead of as targets hurts both levels.