Flexumm improves and speeds unified multimodal models
FlexUMM: Flexible Representation Learning for Unified Multimodal Models
Artificial Intelligence
Summary
Combining different tasks like understanding and generating content in one AI model can create problems because these tasks sometimes need different kinds of thinking. The authors found that sharing everything in the model is too limiting, so they made FlexUMM, which decides when to share parts of the brain and when to keep things separate for each task. This flexibility helps the model work better and faster by avoiding unnecessary work and reducing conflicts between tasks. FlexUMM can be added to many existing models to make them smarter and more efficient.
What this means in practice
- •For ai engineers: Build more accurate and efficient unified models that handle multiple types of inputs and tasks by using adaptive sharing of model components.
- •For cloud service providers: Reduce computational costs by deploying models that dynamically skip unnecessary processing for multimodal AI workloads.
Authors
Yinyi Luo, Haoyue Bai, Yi Ge, Nilton Cesar Rojas Vales, Marios Savvides, Jindong Wang
Abstract
Unified multimodal models (UMMs) can integrate understanding and generation within a single backbone. However, it is commonly known that these two capabilities can cause conflicting representations, leading to a central question: how representation and computation should be shared across functionalities? Existing UMMs often rely on a largely fixed structure, where different inputs and functionalities follow the same depth and parameter-sharing pattern. We argue that this assumption is too restrictive: more effective unification requires sharing the computation that supports cross-functional communication while specializing the capacity that becomes task- or input-specific. We introduce FlexUMM, a flexible UMM framework that dynamically decides when to preserve shared computation, when to route specialized capacity, and when to stop further processing. Evidence shows that this flexible allocation improves UMMs along two complementary axes. It improves accuracy by reducing harmful competition between functionalities, and it improves efficiency by avoiding unnecessary computation once an input representation is sufficiently mature. Beyond simple architectural unification from the pre-trained models, FlexUMM works as a model-agnostic, functional plug-in to make existing UMMs more unified, stronger, and efficient. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/flexumm.