MetaEncoder improves multimodal decision making with language based choices
MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface
Computation and LanguageArtificial Intelligence
Summary
Many computer systems make decisions by picking from a fixed set of options, but these options are usually coded in a strict format. The authors explore a way to represent both the user's question and possible answers entirely in natural language, supported by images and videos. They created MetaEncoder, a system that learns to match questions with answers using a two-part network trained on aligning requests and choices. Their tests show MetaEncoder works well in many tasks, especially when deciding from a large or small number of options, but it struggles with very complex reasoning tasks.
What this means in practice
- •For multimodal ai developers: Implement systems that select correct answers from millions of candidates using natural language descriptions plus images or videos.
- •For search engine engineers: Enhance retrieval systems by aligning natural language queries with large candidate pools including multimodal data.
Authors
Jianpeng Cheng, Guangyu Sun, Aashu Singh, Benyu Zhang, Haixing Dai, Hossein Mansour, Jiangfan Zhang, Shlok Kumar Mishra, Wei Sun, Xuanming Cui, Yanli Liu, Qi Guo, Max Xiangjun Fan, Jun Xiao
Abstract
System One models output constrained decisions and probability distributions rather than free-form text generation. While prevailing paradigms rely on structured schema objects to encode state, intent, and candidate choices, we revisit a fully natural language-based System One interface. In this framework, both the user request and each candidate option are expressed in natural language, supported by multimodal (image and video) auxiliary inputs. We introduce MetaEncoder, which fine-tunes a pre-trained Muse-Glimmer 30B decoder into an instruction-following decision-making encoder. To scale effectively across both small closed-set (< 256) and massive open-set (millions) candidate spaces, MetaEncoder employs a bi-encoder architecture trained via unidirectional contrastive learning for request-candidate alignment. We conduct extensive evaluations across 11 benchmark suites and 190 tasks spanning multimodal decision-making, understanding (closed-set) and retrieval (open-set), highlighting where MetaEncoder beats SOTA multimodal encoders, as well as its current limits on reasoning-intensive tasks.