Human centered graphs improve video question answering accuracy
THG-Omni: Traceable Human-Centric Graph Supervision for Video Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Videos often have multiple people and events, but models that answer questions about them usually don’t show which parts in the video support their answers. The authors created THG-Omni, a system that connects detailed information about people and actions in videos to questions and answers, teaching the model how to use this info to explain its reasoning. This approach helps the model better understand social situations, emotions, and deception in videos. Experiments showed that using these detailed graphs leads to more accurate and interpretable answers.
What this means in practice
- •For video content analysts: Improve automated interpretation of social and emotional cues in videos by using structured participant and event information to support answers.
- •For security and surveillance teams: Enhance detection and explanation of deceptive behavior in video footage by leveraging traceable human-centric graphs for reasoning.
Authors
Jisheng Dang, Kai Wei, Ronghao Lin, Bimei Wang, Hong Peng, Wei-Shi Zheng, Bin Hu, Qi Tian, Tat-Seng Chua
Abstract
Omni-modal models can answer questions about video, but their predictions rarely show which participant, event, or interval supports an answer. Moreover, answer-level supervision alone does not teach a model to interpret structured video evidence. To systematically mitigate this issue, we present THG-Omni, a human-centric graph-supervision framework that links Segment Anything Model~3 participant tracks with temporally grounded Qwen3-VL annotations. Candidate semantic claims are checked against the source video and available audio before they are used to construct training questions, while track geometry provides explicit spatial and temporal supervision. The resulting questions teach an omni-modal reasoning model to interpret participant attributes, actions, affect, interactions, and their timing from a structured graph. At inference, THG-Omni uses a predicted graph constructed from the evaluation video. Experiments on IntentBench show consistent improvements over the unadapted backbone across social reasoning, emotion understanding, and deception perception. On NExT-QA, THG-Omni achieves 89.54\% accuracy with the matched graph, compared with 85.10\% without graph input. Further validation experiments show that compact graph input achieves 78.22\% accuracy, compared with 78.02\% using the full graph and 77.58\% without graph input. A complementary diagnostic demonstrates that the predicted graph preserves evidence useful for answering questions even without the source video and audio. Together, these findings show that THG-Omni improves human-centric video reasoning while grounding its evidence in identifiable participants and events. Our codes are available at https://anonymous.4open.science/r/review-n94hhfcwp5666/