EgosIS improves drone video understanding by separating motion from scene changes
EgoSIS: From Factorized Visual Ego-Transitions to Motion-Canonical Spatial Evidence for UAV Reasoning
Computer Vision and Pattern Recognition
Summary
When drones capture video, it can be hard for computers to tell if something moved or if the camera itself moved. The authors created EgoSIS, a tool that helps separate the drone's motion from changes happening in the scene using only regular video data without needing extra sensors. EgoSIS processes how images shift and remembers the drone’s recent movement to provide clearer visual clues for understanding the environment. This improves answering questions about drone videos by making it easier to know what the drone is seeing versus how it is moving.
What this means in practice
- •For drone software developers: Improve drone video analysis systems by better separating drone motion from scene changes using only RGB video input.
- •For autonomous vehicle teams: Enhance perception modules by incorporating interpretable motion-canonical visual cues for reasoning about ego-motion and scene dynamics.
Authors
Jingpu Yang, Fengxian Ji, Mingxuan Cui, Yilin Sun, Hang Zhang, Jianhua Zhu, Yufeng Wang
Abstract
UAV video question answering requires separating camera motion from changes in the scene, but RGB-only multimodal models receive no explicit, stable reference for that separation. We present EgoSIS, a pose-free adapter that converts RGB-derived bidirectional flow into motion-canonical visual evidence in three stages. Factorized Visual Ego-Transitions (FVET) fits a robust image-plane transition and exposes motion, residual-support, and reliability factors. Reliability-Gated Ego-Transition Memory (ReTEM) uses reliability-weighted updates for a bounded history and re-anchors it at cuts or sustained uncertainty. Ego-Aligned Spatial Evidence (EASE) warps supported visual features into each segment's local anchor and injects four spatial evidence tokens per visual slice through zero-initialized residuals, without changing Qwen's visual-token count. On SIS-Bench, EgoSIS-8B obtains 89.9\% perception, 82.5\% perception-plus-memory, and 76.2\% overall accuracy, with the largest gains concentrated in self-awareness perception and memory. The adapter thus provides an interpretable interface between optical flow and spatial reasoning.