Summary
Videos taken by drones can be hard for computers to understand because the camera is always moving, making it difficult to tell if something in the scene changed or if the change is just from the camera moving. The paper presents a method called EgoSIS that helps separate the drone’s own movement from what is actually happening in the scene, without needing extra sensors. The authors break down the visual changes into parts like motion and reliability, keep track of this information over time, and use it to improve the computer's ability to understand and answer questions about the video. This approach shows better accuracy in recognizing movement and memory tasks on a test for drone video understanding.
UAVvideo question answeringcamera motionoptical flowmotion canonicalizationspatial reasoningvisual featuresmemory updatevisual ego-transitionsscene understanding
Authors
Jingpu Yang, Fengxian Ji, Mingxuan Cui, Yilin Sun, Hang Zhang, Jianhua Zhu, Yufeng Wang
Abstract
UAV video question answering requires separating camera motion from changes in the scene, but RGB-only multimodal models receive no explicit, stable reference for that separation. We present EgoSIS, a pose-free adapter that converts RGB-derived bidirectional flow into motion-canonical visual evidence in three stages. Factorized Visual Ego-Transitions (FVET) fits a robust image-plane transition and exposes motion, residual-support, and reliability factors. Reliability-Gated Ego-Transition Memory (ReTEM) uses reliability-weighted updates for a bounded history and re-anchors it at cuts or sustained uncertainty. Ego-Aligned Spatial Evidence (EASE) warps supported visual features into each segment's local anchor and injects four spatial evidence tokens per visual slice through zero-initialized residuals, without changing Qwen's visual-token count. On SIS-Bench, EgoSIS-8B obtains 89.9\% perception, 82.5\% perception-plus-memory, and 76.2\% overall accuracy, with the largest gains concentrated in self-awareness perception and memory. The adapter thus provides an interpretable interface between optical flow and spatial reasoning.