Tracking points beyond camera view using 360 degree video expansion
TAP-Out: Tracking Any Point in 360° via Outpainting
Computer Vision and Pattern Recognition
Summary
Tracking objects when they move outside the camera’s view is very challenging because we can’t see what happens next. The authors created TAP-Out, a method that predicts what the whole 360-degree scene looks like beyond the camera’s narrow view. It then finds and tracks objects in this wider panorama, letting regular tracking tools follow objects that leave the original view. This approach works better than previous methods and can also segment or detect objects that go out of frame.
What this means in practice
- •For video surveillance teams: Track objects continuously even when they leave the camera’s original field of view using expanded 360-degree scene modeling.
- •For virtual reality developers: Create seamless 360-degree object tracking in VR environments where users' views frequently shift and objects move outside of narrow camera views.
Authors
Finlay G. C. Hudson, James A. D. Gardner, William A. P. Smith
Abstract
Tracking points that leave the camera's field of view (FoV) requires reasoning about unobserved scene dynamics, a fundamentally different problem from tracking visible correspondences. We present TAP-Out, an offline method that makes this out-of-view (OOV) reasoning explicit by leveraging generative video outpainting as a world-model prior. Given a narrow-FoV input video, TAP-Out outpaints a complete 360° equirectangular video, locates target objects within the panoramic scene via video object segmentation, and synthesises object-centred perspective crops on which standard in-frame trackers operate. This decoupled design composes off-the-shelf foundation models without requiring task-specific architectural modifications. On the TAPVid360 benchmark, TAP-Out achieves state-of-the-art performance, improving OOV tracking accuracy by approximately 3x over the best prior method. We demonstrate its flexibility on out-of-frame object segmentation and detection and we provide detailed diagnostic analyses across motion regimes, static versus dynamic content, and hallucination quality, characterising when and why the approach succeeds or fails.