Streaming video model starts conversations without prompts

AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Most AI helpers wait to be asked questions before answering, missing chances to talk when something interesting happens on screen. The authors designed AURA 2.0, a system that can notice important events in videos and start conversations on its own. It can also use extra tools and clearly link its answers to objects shown in the video. This helps the system keep track of long videos and chat more naturally over time. The authors tested their system and made the code available for others to use.

What this means in practice

  • For video content creators: Add an AI that actively comments and suggests discussion topics during live video streams without waiting for viewer prompts.$Commercial implications: It enables interactive streaming services and tools that engage audiences with AI-driven conversations, enhancing viewer experience and retention.
  • For surveillance operators: Deploy AI that autonomously points out and discusses events detected in ongoing video feeds to support monitoring and decision-making.

Authors

Shuhan Li, Huankang Guan, Xintong Guo, Yang Bo, Yuyang He, Dunyuan Xu, Jingyan Chen, Fang Liu, Tsz Kin Wan, Jinpeng Chen, Zhenqi He, Yuxuan Hu, Wenbo Li, Han Gao, Qijun Li, Long Chen, Hongsheng Li, Rui Liu

Abstract

Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.