Video models unified to handle diverse tasks without retraining
Video, Ergo Genero: Unifying Video Tasks via Spatiotemporal Analogy
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Video models usually need special training for each new task, which can be slow and costly. The authors created a method called ViGeo that lets a video model solve many different tasks by using examples shown during the process, without extra training. This approach works even on types of video data the method hasn't seen before, like from event cameras. They also found a way to fix a problem where the model guesses the task based on the example format, rather than the example content.
What this means in practice
- •For video software developers: Create video applications that perform new tasks without collecting training data or retraining models.
- •For surveillance system engineers: Adapt video processing systems to new sensor types and tasks without additional model updates.
Authors
Chia-Hsiang Kao, Belinda Zeng, Bharath Hariharan, Menglin Jia
Abstract
Adapting video models to new tasks typically requires dedicated data curation and fine-tuning. While visual analogy provides a training-free alternative by specifying tasks in-context, it remains restricted to the image domain. To explore whether analogy-based methods can unify diverse video tasks and generalize to out-of-distribution scenarios, we introduce ViGeo, a framework that extends visual in-context learning to the video domain via spatiotemporal canvas completion. Evaluated on a diverse task taxonomy with a strict train-test split, ViGeo generalizes to unseen video manipulations and zero-shot modalities (e.g., event cameras). Finally, we identify task internalization, where a query format associated with a pretrained task overrides the demonstration, and show that this shortcut can be removed with a small amount of task-unrelated data, highlighting the need to decorrelate prompt format from task identity.