Multimodal generation made simple across images speech and proteins
Multimodal Co-Generation With Continuous Bitstream Diffusion
Computation and Language
Summary
Generating content that mixes different types of data like images, speech, and proteins is usually complicated with different rules for each. The authors show that a single method called continuous bitstream diffusion (CoBit) can handle all these types together using the same approach. They trained separate models from scratch but all used the same basic setup and could work jointly or conditionally depending on the input. Their tests showed strong results in generating and understanding images, speech with text, and protein shapes with sequences, all without needing special tricks for each type.
What this means in practice
- •For speech technology teams: Build speech-text models that synthesize and transcribe speech jointly from a single model without separate pretraining.
- •For protein design engineers: Generate protein sequences and structures jointly from one model to support protein folding and inverse folding tasks.
Authors
Georgios Batzolis, Georgios Kontos, Bruno Trentini, Pranam Chatterjee, Micha Livne, Mark Girolami, Luca Ambrogioni
Abstract
We show that continuous bitstream diffusion (CoBit), originally developed for language modeling, provides a unified framework for joint and conditional multimodal generation using the same training and sampling protocol. Across image-text, speech-text and protein sequence-structure modeling, models are trained separately from scratch, but each uses a single input projection, transformer backbone, bitwise prediction head and binary score-matching objective shared across its modalities. Within each setting, a single checkpoint supports joint generation and the conditional directions included in training by changing which parts of the bitstream are observed. We first validate the approach on MNIST-Sum, a controlled image-text task pairing raw binary images with equations encoded as text, achieving near-perfect conditional accuracy and high joint consistency without learned tokenizers. We then apply the same protocol at larger scale to speech-text and protein sequence-structure modeling. Trained without text pretraining, a single speech-text model performs speech synthesis, recognition, continuation, and unconditional co-generation. Jointly generated speech and text achieve 3.5% word error rate between the generated text and automatic transcriptions of the generated speech. In protein modeling, a single CoBit checkpoint supports forward folding, inverse folding, and joint sequence-structure generation. For jointly generated proteins spanning 100-500 residues, computational refolding of CoBit's sequences yields structures that agree with the generated backbones at a mean self-consistency TM-score of 0.911. Together, these results show that CoBit provides a common generative framework for substantially different modalities and forms of cross-modal dependence, without introducing a domain-specific objective or sampling procedure.