Pixelumm unifies image and video understanding and generation in pixels
PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
Computer Vision and Pattern Recognition
Summary
Understanding and creating images and videos usually require different methods, which can be complex and slow. The authors present PixelUMM, a new model that handles both images and videos directly using raw pixel data, without needing complex encoding steps. This model uses a unique approach to break down pixels spatially and temporally, allowing it to understand and generate visual content in a unified way. Their tests show PixelUMM works well across multiple tasks involving images and videos.
What this means in practice
- •For computer vision engineers: Develop systems that simultaneously interpret and create images and videos without separate visual encoding steps.
- •For multimedia content developers: Generate consistent and coherent video clips from textual or visual prompts using a unified pixel-based approach.
Authors
Cong Wei, Xuanchi Ren, Bryan Chu, Weiming Ren, Huan Ling, Jiahui Huang, Laura Leal-Taixé, Sanja Fidler, Wenhu Chen, Zian Wang, Jay Zhangjie Wu
Abstract
Unified Multimodal Models (UMMs) often rely on separate visual representations for understanding and generation, increasing visual context length and complicating integration with established vision-language pretraining pipelines. Recent advances in pixel-space modeling offer an encoder-free alternative, but extending this paradigm from images to videos is non-trivial: video understanding and generation adopt different temporal representations, leaving the design of a unified visual interface an open question. We present PixelUMM, an encoder-free model for unified image and video understanding and generation directly in pixel space. PixelUMM represents images as spatial patches and videos as spatiotemporal tubelets, connecting raw pixels to a shared multimodal backbone through single-layer linear projections. Its Mixture-of-Transformers architecture combines shared attention with task-specific parameters and extends clean-pixel prediction to video generation, jointly supporting autoregressive text prediction and pixel-space flow matching. Experiments show that PixelUMM achieves competitive performance across image and video understanding and generation tasks. We further conduct empirical studies of key design choices, including decoder design and spatial-temporal patch size, providing insights for future pixel-space unified multimodal models.