Existing omnimodal AI assistants struggle to help reliably in daily life
Omni-Life: Benchmarking Omnimodal Real-Time AI Assistants in People's Daily Lives
Computation and Language
Summary
AI assistants that can understand sound and vision together are meant to help people in everyday situations. The authors created a new test called Omni-Life to check if these assistants respond correctly and on time, especially in tricky or dangerous moments. They found that current AI models rarely get everything right, sometimes responding too late or when no help is needed. Their work shows it's still hard for these AIs to know exactly when to speak or stay silent in real-world settings.
What this means in practice
- •For real-time ai developers: Use Omni-Life to test and improve AI assistants’ ability to recognize events and decide when to respond promptly yet correctly in daily life scenarios.
- •For smart home device engineers: Design smart devices that optimize response timing and reduce false alarms by applying findings about reminder frequency and latency tolerance from the benchmark.
Authors
Mengze Hong, Ziwei Shang, Di Liang, Zijian Li, Liang Zhu, Hongyu Shan, Robin Zhou, Minlong Peng, Zhiqiang Kou, Jia Wang, Di Jiang, Chen Jason Zhang
Abstract
Real-time omnimodal assistants are expected to support people in everyday life, but audiovisual understanding alone does not establish whether they can provide timely and appropriate help. We introduce Omni-Life, a benchmark grounded in real-world user needs, deployment scenarios, and device-dependent observations. It combines controlled video synthesis for hazardous situations and edge cases with annotated real-world videos under a shared temporal protocol. The benchmark contains 400 videos: 290 requiring assistance, with 555 required intervention events, and 110 requiring silence throughout to test whether models can withhold unnecessary responses when no assistance is needed. We evaluate twelve configurations featuring mainstream real-time omnimodal models, jointly assessing event coverage, response content, timing, and unwanted output. At most 4.48% of videos requiring assistance are handled completely correctly. To examine practical deployment strategies, we study task reminders and response-latency tolerance. Reminders can improve recall by 17.12 percentage points but add approximately 4.6 unwanted responses per minute of reference silence. Increasing latency tolerance by 10s improves recall by at most 5.95 percentage points. These results show that existing models remain insufficient to deliver reliable assistance in real-life applications, calling for tailored model optimization in instruction following, timely proactive intervention, and withholding responses when assistance is not needed.