Unlocking Motion in Expressions: Temporal Calibration for Referring Video Object Segmentation
2026-08-17 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors focus on a task where a system must find and outline specific objects in videos based on spoken or written descriptions. They noticed that current methods don't properly link the movements mentioned in the descriptions with how motion information is used in the videos. To fix this, they created a new approach that reads the language for clues about motion, then carefully adjusts how motion data helps identify the object over time. Their method also organizes video frames into meaningful time segments related to the description, making the process more efficient. They tested their approach on multiple video datasets and showed it performs better than previous methods.
Referring Video Object Segmentationmotion semanticscross-modal temporal modelingnatural language descriptionMotion Signal ProcessingMotion Influence CalibrationSemantic Temporal Stage Constructionvideo segmentationtemporal decision making
Authors
Yiwen Jiang, Zhengtong Zhu, Ruixin Zhang, Jiaqing Fan
Abstract
Referring Video Object Segmentation (RVOS) aims to segment referred objects at the pixel level in video sequences based on natural language descriptions. Existing methods typically introduce motion information within a unified cross-modal temporal modeling framework, where language cues are used for target localization and segmentation. However, the dependency of expressions on motion semantics is not explicitly modeled, making it difficult to adaptively adjust the use of motion information according to different semantic requirements. To address these issues, we propose an Expression-driven Motion Calibration (EMC) framework for RVOS that explicitly unlocks and leverages the motion semantics within expressions. The proposed method extracts interpretable motion control signals from expressions via a Motion Signal Processing (MSP) module, and employs a Motion Influence Calibration (MIC) module to adjust the contribution of motion cues during temporal decision making. In addition, a Semantic Temporal Stage Construction (STSC) module is introduced to build expression-relevant temporal stages, providing a compact temporal candidate space for motion calibration. Through extensive evaluation on six standard benchmarks, including Ref-YouTubeVOS, Ref-DAVIS17, MeViS (valid/valid$^u$), A2D-Sentences, and JHMDB-Sentences, the superiority of our method is validated. We will release the code on https://github.com/Jeven7/EMC.