Openhome measures smart-home camera video understanding by vision language models
OpenHome: From Perception to Insight in Smart-Home Camera Videos
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Smart-home security cameras record lots of videos, but these videos can be hard for AI models to understand because they lack clear narration or subtitles and vary a lot in lighting and angles. The authors created OpenHome, a special test with questions about real smart-home videos, to see how well vision language models can interpret what is happening. They tested many existing models and found that the best ones do well on simple perception tasks but struggle more when reasoning or understanding intent and appropriate responses is required. This helps show where these models need improvement to be more helpful in smart-home settings.
What this means in practice
- •For smart-home device makers: Assess and improve the ability of AI models to understand security camera videos for better automatic alerts and insights.$Commercial implications: Enables development of advanced AI-powered smart-home camera products that provide meaningful video insights to consumers.
- •For video analytics developers: Use the OpenHome benchmark to test and refine algorithms for video perception, reasoning, and intent detection in home environments.
Authors
Xianglong Wang, Neil Rodrigues, Pei Guo, Yiru Fang, Kevin Beussman, Shuyang Wang, Jingzhi Wang, Liangju Li, Shunyao Xu, Zhongwei Cheng, Wei Li, Lin Chen
Abstract
Vision language models (VLMs) have been increasingly adopted in consumer smart-home camera systems to interpret recorded videos and provide insights. However, video clips captured from smart-home cameras are different from curated web videos dominating existing VLM benchmarks. Smart-home video clips often do not contain subtitles or narration and are captured from different viewing angles, under various lighting conditions, and in diverse scenarios. However, the capabilities of the model in this domain, under sparse visual signals and limited context, are not fully assessed by existing VLM benchmarks. To close this gap, we introduce OpenHome, a visual-only, four-option multiple-choice benchmark for evaluating smart-home video understanding. It contains 2,254 questions on 1,118 real-world video clips and organizes 10 tasks into three capability levels. Perception evaluates visible content, Reasoning evaluates causal, spatial, and temporal relationships, and Insight evaluates subject intent, supporting evidence and appropriate homeowner responses. Based on the dataset, we describe the task design, data pipeline and evaluation protocol, and report results for 32 closed-source and open-weight VLMs. The best model, Gemini 4 Argon, reaches 81.2% overall accuracy. Frontier closed-source models outperform open-weight models on every level. The best closed-source model leads the best open-weight model by 16.3% on Perception, 22.4% on Reasoning and 37.4% on Insight.