Moving regions identified precisely under camera motion in videos

Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

Computer Vision and Pattern Recognition

Summary

When a camera moves, it’s hard to tell which parts of a video are really moving objects versus just changes caused by the camera itself. The authors developed MoRSe, a method that compares two video frames together to find the exact moving parts, not just whole objects that might only partially move. This method avoids common mistakes and works well even when nothing in the scene moves. They also created a benchmark dataset called MoRSe-5K to test these abilities under various motion conditions.

What this means in practice

  • For video editing teams: Automatically detect and isolate truly moving regions in video footage for editing that adapts to camera motion.
  • For surveillance operators: Improve detection accuracy by distinguishing actual object motion from camera-induced apparent motion in security videos.

Authors

Wenhua Jin, Tiancheng Chen, Sheng Tang, Jiahao Li, Yupeng Liu, Weiqi Zhang, Juntong Fang, Zesong Li

Abstract

Moving object segmentation is conventionally formulated at the sequence and object level, marking an entire instance as moving once it moves anywhere in a video. Recent instantaneous formulations distinguish moving from stationary intervals, but still generate masks at the object level through frozen segmentation foundation models. Because the mask is derived from an instance-level proposal, motion is typically attributed to the entire instance rather than localized to the regions that actually move. Moreover, such instance-level predictions can produce masks even when no part of the instance is actually moving, revealing a more fundamental issue: motion is not an intrinsic property of an instance; it can only be determined by comparing at least two frames. Building on this premise, we introduce MoRSe, which jointly interprets two frames to predict a moving-region mask, distinguishing actual scene motion from apparent motion induced by the camera. A single RGB encoder replaces multi-stage foundation stacks, so no optical flow, point tracks, or separate mask generator is needed at inference. We release MoRSe-5K, a benchmark of 42 sequences with Static, Dynamic-Static, and Dynamic settings designed to evaluate the model's ability to distinguish and localize moving and stationary regions under different camera and scene motion conditions. MoRSe outperforms both specialized moving object segmentation methods and dense 4D reconstruction methods with motion-thresholded predictions, while producing the fewest false positives on sequences where nothing moves.