Realtime benchmarks reveal limits of AI in fast changing tasks
Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?
Artificial Intelligence
Summary
Realtime-Bench tests how well AI models handle tasks where speed and quick decision-making matter. The authors show that these models often fail to act fast enough in changing situations, with success rates below 50%. They introduce different timing tests to separate delays caused by hardware from how well the AI thinks about time. The research finds that taking more time to reason doesn’t always help, revealing a balance between fast reaction and careful thinking. These insights point to ways to improve AI that acts continuously and adapts its thinking speed based on real-time demands.
What this means in practice
- •For interactive ai developers: Build AI systems that maintain effectiveness when required to react instantly to changing environments.
- •For robotics engineers: Design robots that balance decision accuracy with speed to succeed in real-time physical tasks.
Authors
Jingyu Tang, Xu Yang, Zhehao Zhang, Zheng Wang, Qi Cheng, Haozhe Zhao, Manling Li, Wenxuan Li, Xide Xia, Jiaxin Pei
Abstract
Real-time interaction requires agents to perceive evolving environments and act under temporal constraints, yet existing evaluations of real-time streaming models focus on conversational quality and streaming perception, leaving effective action under strict temporal constraints untested. We introduce Realtime-Bench, a suite of carefully curated dynamic tasks where delayed actions directly affect success. Under Wall Clock, where environments advance according to actual inference latency, the best success rate reaches only 44% and the best Pass^3 is only 36%, demonstrating that real-time interaction remains challenging. To separate serving latency from models' real-time streaming capability, we compare Wall Clock with Paused Clock, which removes inference delays as an idealized reference, and introduce Token Clock, which advances simulated time according to generated token counts, testing temporal awareness independently of physical serving speed. We find that temporal costs limit success in evolving environments and that higher reasoning effort has mixed effects under time pressure, reflecting a trade-off between decision quality and timely execution. These findings suggest directions for improving real-time multimodal agents, including adaptive reasoning with temporal awareness and continuous perception and action.