RobotEQ-Video benchmarks social intelligence in proactive robots
RobotEQ-Video: A Video-Centric Benchmark for Social Proactive Intelligence with World-State Taxonomy
Computer Vision and Pattern RecognitionHuman-Computer Interaction
Summary
Helping robots understand not just what task to do but how to act socially is hard, especially when only still images are used. The authors created a large video-based benchmark called RobotEQ-Video that shows many real-life situations and labels how appropriate a robot's behavior would be in these social settings. They organized the scenarios with a detailed system describing human and environment states. Tests with current AI systems show they still struggle compared to humans. This work helps improve robots’ social awareness by focusing on videos, which have more clues than pictures.
What this means in practice
- •For robotics developers: Evaluate and improve social intelligence models for robots using a comprehensive video benchmark reflecting diverse real-world situations.
- •For human-computer interaction teams: Assess how their systems understand and respond to human social cues over time with a large annotated video dataset.
Authors
Xinyi Che, Zheng Lian, Kuofei Fang, Xuehao Wang, Xinghai Gao, Junqing Wu, Chuyu Wu, Liyi Liu, Yanhan Huang, Keyi Xie, Haomin Ouyang, Jinyang Wu, Fan Zhang, Runhao Zeng, Xun Yang, Bin He
Abstract
Social Proactive Intelligence (SPI) extends proactive assistance beyond task completeness to consider social appropriateness in diverse embodied scenarios. However, prior SPI research faces two key limitations. First, existing work focuses on static images, whereas dynamic videos provide crucial cues for inferring human states and needs, offering richer information than isolated images. Second, prior work often relies on free-form data collection pipelines, which fail to guarantee comprehensive coverage of diverse scenarios. To address these gaps, we introduce RobotEQ-Video, shifting the focus from image-centric to video-centric analysis. To ensure comprehensive video coverage, we construct a hierarchical world-state taxonomy organized into a four-level coarse-to-fine structure, comprising 6 domains, 20 dimensions, 142 level-1 attributes, and 816 level-2 attributes. The resulting benchmark comprises 2K+ videos with 100K+ human annotations and 16K+ labels for assessing behavior properness. Benchmark evaluation reveals that current systems remain unreliable and fall short of human performance. We further explore how world models can help tackle this task. This work advances SPI research from static images to dynamic videos and ensures more comprehensive scenario coverage during benchmarking.