Multimodal model improves real time detection of camera wearer conversations

Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

Computer Vision and Pattern RecognitionArtificial Intelligence

Summary

Understanding who is talking to someone wearing a camera is important for real-time social interactions but has been hard to do well before. The authors rethought this by making the task about recognizing who is speaking every moment instead of just judging whole clips after the fact. They also included different talking situations, like talking to others or talking to oneself, rather than lumping all non-direct speech together. They created a large new dataset with detailed labels and built a new model that combines sound, visuals, and the meaning of speech. Their model detects when someone is talking to the camera wearer more accurately than previous methods.

What this means in practice

  • For wearable device developers: Improve real-time interaction features in smart glasses by accurately detecting when others talk to the wearer using multimodal input.$Commercial implications: Enables smart glasses to deliver context-aware responses by detecting direct speech, enhancing user interaction and experience.
  • For video analytics software teams: Enhance live analysis of egocentric video streams to label social speaking states, distinguishing between who speech is directed to.

Authors

Feiyu Du, Xi He, Jia Li, Yapeng Tian, Weili Wu

Abstract

Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM) studies are commonly formulated as offline clip-level recognition, which is poorly aligned with online interaction and overlooks the diverse non-TTM speaking states that naturally arise in egocentric videos. In this paper, we revisit this problem by reformulating it as an online, frame-level prediction task. Instead of treating TTM as a binary problem against a single negative class, we model it in the presence of diverse and previously underexplored non-TTM states, such as talking-to-others, self-talking, and background conditions. To support this new formulation, we construct an Online TTM Dataset consisting of 406 egocentric video clips with approximately 900K annotated frames, each labeled with frame-level social interaction categories (e.g., background, TTM, talking-to-others, self-talking), by extending the Ego4D social interaction benchmark. In this benchmark, we evaluate five adapted baselines and develop a new model that integrates social cues across modalities. Experimental results show that our multimodal model, which jointly leverages audio, visual, and speech-semantic cues, achieves 75.5% frame-level F1 on TTM, outperforming strong baselines and enabling a systematic analysis of how different speaking states affect TTM recognition.
View PDFOpen arXiv