From Modalities to Propositions: A Language-Centric Framework for Multimodal Intelligence
Artificial IntelligenceComputer Vision and Pattern RecognitionMachine LearningMultimedia
Summary
The authors suggest a way to describe different types of data like images, videos, and text using simple statements about what is happening or shown. They create a shared list of these statements so all kinds of data can be understood in the same way. This helps computers connect and reason about information from different sources, like recognizing actions or objects across formats. They tested this idea with examples from self-driving cars and other real-world data.
Authors
Nadine Chang, Maying Shen, Shizhe Diao, Jialiang Wang, Jingde Chen, Thomas Breuel, Pavlo Molchanov, Rafid Mahmood, Jose M. Alvarez
Abstract
We propose a language representation for multimodal data in which any observation, whether image, video, or text, is expressed as a bag of atomic propositions, simple statements about the entities, actions, and relations in a scene. A global semantic codebook unifies these into a shared vocabulary of canonical atomic propositions, placing every modality and observation into one interpretable space that spans fine grained facts to high level concepts and composes into richer ones. This brings interpretability with reasoning, cross-modal understanding and retrieval, and compositionality that enables complex multimodal understanding, rich data curation and complex structured retrieval. We demonstrate the framework on autonomous driving and open-world data.