Fovea-JEPA improves camera motion recognition by fixing geometric blind spots
Fovea-JEPA: Overcoming Geometric Blind Spots in V-JEPA for Camera Motion Recognition
Computer Vision and Pattern Recognition
Summary
Recognizing how a camera moves—whether it turns, zooms, or shifts—is tricky because the changes in what the camera sees can be confused with things moving in the scene. The paper shows that current AI models struggle to use geometric clues well in some parts of the images. To fix this, the authors created Fovea-JEPA, which finds these weak spots and adds geometric information from another specialized model. This combination helps better tell how the camera moves, and tests show it works much better than previous methods.
What this means in practice
- •For video analytics developers: Develop video systems that more accurately detect camera movement types by incorporating geometric features to disambiguate camera and object motions.
- •For augmented reality engineers: Integrate enhanced camera motion recognition to improve scene understanding and alignment when the camera moves with complex rotations and zooms.
Authors
Xiangkun Shi, Hanhui Li, Shiyue Zhang, Chang Liu, Junwen Pan, Zhenbang Sun, Xiaodan Liang
Abstract
Camera motion recognition requires separating viewpoint changes caused by camera translation, rotation, or zoom from object motion and scene dynamics. Video-language models often rely on semantic cues and struggle with the geometric evidence needed for this task. Joint-Embedding Predictive Architectures such as V-JEPA offer strong spatio-temporal representations, but our experiments suggest that they underrepresent geometric cues in some regions. Under temporal-reversal supervision, V-JEPA improves overall but degrades on several motion families, whereas a weaker geometry-oriented VGGT predictor improves across all families. We refer to these geometrically underrepresented regions as geometric blind spots and propose Fovea-JEPA to identify and address them. Using V-JEPA as the base model and VGGT as a geometry expert, we construct intervention pseudo-labels from their prediction errors to weakly supervise a blind-spot predictor. The predictor produces token-wise response maps, which are aggregated into a clip-level intervention score to guide adaptive fusion of visual and geometric features. This selective intervention introduces complementary geometric information while preserving reliable visual predictions. Experiments on CameraMotionDataset, CameraBench, and ShotBench demonstrate the effectiveness of Fovea-JEPA, with gains of 9.67, 8.67, and 24.38 percentage points over previously reported state-of-the-art baselines, respectively. Code and models will be released upon acceptance.