Papers for

multimedia app developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

UniMore improves image and text understanding with fine-grained training

UniMore: Fine-Grained Supervision for Unified Understanding and Generation

Abstract: Unified multimodal models combine image understanding and generation within one system, creating an opportunity to improve both capabilities through paired image--text supervision. We examine the outputs of a pretrained unified model in both directions and observe a loss of specificity: generated images can contain the requested objects with incorrect attributes, while captions identify a scene without expressing its distinguishing details. These failures call for supervision that specifies both the visual content to realize and the descriptive differences to distinguish. We propose UniMore, a post-training method that derives these complementary learning tasks from the fine-grained content of image--text pairs. The generation route predicts grounded crops from their corresponding phrases, matching the scope of the condition and visual target while using the native visual objective. The understanding route combines positive-caption fitting with ranking against captions that omit or alter details, conditioned on the full image. The two routes are optimized independently, and their learned updates are combined into a single unified model without changing its inference procedure. Experiments show gains across multiple understanding and generation benchmarks.

Fri 6 NovMultimediaComputer Vision and Pattern Recognition
The gist
Unified models that do both image recognition and image creation often miss details either when describing a picture or making one from a description. The paper finds these models lose important specifics like the right color or unique details. The authors introduce UniMore, a training approach that teaches the model to pay attention to precise parts of images and their matching words. This improves both the quality of generated images and the accuracy of image captions without changing how the model works when it is actually used.
Open → 2611.08996v1

Multimodal learning method captures unique and shared data features

Multimodal Information Interactions Bottleneck for Multimodal Self-Supervised Learning

Abstract: Multimodal self-supervised learning is challenged by heterogeneity, unequal informativeness, and complex information interactions across multimodal data sources such as images, text, audio, and tabular data. While contrastive methods effectively align shared semantics, they typically rely on strong multi-view redundancy assumptions, overlooking modality-specific and synergistic information, thus leading to suboptimal representations. We propose the Multimodal Information Interactions Bottleneck (MIIB), a principled framework grounded in Partial Information Decomposition that explicitly encodes redundant, unique, and synergistic information within multimodal interactions. MIIB employs a dual-path model architecture for modality-specific encoding and multimodal fusion, optimized using surrogate tractable entropy-based objectives that enforce minimal sufficient representations without requiring explicit mutual information estimation. By leveraging variational bounds and score-based entropy estimation, MIIB efficiently captures complex cross-modal interactions while suppressing superfluous information. Experiments across diverse benchmarks demonstrate consistent improvements over state-of-the-art multimodal self-supervised methods. Code is available at https://github.com/wanglingzaishi/MIIB.

Thu 5 NovComputer Vision and Pattern RecognitionMachine Learning
The gist
Multimodal self-supervised learning faces the challenge of understanding different types of data like images, text, and audio together. The authors propose a new method called MIIB that separates what information is shared across data types, what is unique to each type, and what emerges only when combining them. This new approach helps create better data representations by focusing only on useful information and ignoring noise. They tested their method on various tasks and found it improved performance compared to earlier techniques.
Open → 2611.07718v1

Text guided experts improve video sentiment analysis efficiency

Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis

Abstract: Multimodal sentiment analysis (MSA) aims to understand user-generated videos by integrating multiple modalities. Existing methods often treat modalities as symmetric, overlooking the fact that language provides a dominant semantic prior while nonverbal signals are complementary and noise-prone. To address this, we propose a Text-Anchored Modality Experts with Uncertainty-Guided Routing, termed TAME, that reformulates MSA as an uncertainty-guided refinement paradigm, where the language prior is calibrated and enhanced by selectively trusted nonverbal evidence. Specifically, TAME adopts an asymmetric Mixture-of-Modality-Experts framework, in which textual features act as a fixed and always-activated shared expert, while audio-textual and video-textual cross-modal branches serve as dynamically routed auxiliary experts. Furthermore, we introduce lightweight Prophet Heads to quantify the sample-level predictive uncertainty of each individual modality, and this uncertainty metric is further incorporated into the modality router to realize reliability-aware expert aggregation and fusion. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets demonstrate that TAME achieves state-of-the-art performance on the majority of metrics while reducing parameter count by up to 78.7% and FLOPs by up to 41.2% over the mixture-of-expert baseline. %These results confirm that asymmetric expert construction with uncertainty-guided routing is an effective paradigm for MSA.

Tue 20 OctComputer Vision and Pattern Recognition
The gist
Understanding feelings in videos is tricky because people use words, sounds, and facial expressions together, but words usually tell us the main message. The paper introduces a new method that trusts words most and carefully uses sound and video clues only when they seem reliable, making the system smarter and simpler. The authors use a way to measure how sure each clue is and combine them in a way that gives better results with less computing power. Tests on popular datasets show this method beats others on most measures while being much faster and lighter.
Open → 2610.27571v1

Referring audio visual segmentation avoids overthinking with adaptive reasoning budgets

To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation

Abstract: Multimodal reasoning systems increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding. We show that this assumption fails in referring audio-visual segmentation (Ref-AVS): simple queries can be harmed by excessive reasoning, creating an "overthinking trap" associated with phrase and localization drift, while genuinely ambiguous queries still benefit from multi-step reasoning. To systematically study this, we construct counterfactual reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly decodable information predictive of whether additional reasoning will improve segmentation. A linear probe predicts binary reasoning need with 70.1% accuracy, substantially outperforming text-only features and uncertainty-based proxy signals. As a router, the onset-state controller improves over downstream confidence routing and closely matches Short-then-Decide while using far fewer tokens. Proxy comparisons show that the onset representation contains predictive information not fully recovered by the evaluated hand-crafted features. Reusing this onset state for lightweight, single forward-pass routing reduces autoregressive token consumption by 60% while retaining about 96% of always-long performance. It further provides a cost-quality trade-off on $R^2$-AVSBench without target-label tuning, rather than an accuracy improvement over always-long. Error analysis links harmful long reasoning to phrase and localization drift, illustrating the benefit of bypassing unnecessary reasoning on simple queries. Ultimately, our results support efficient adaptive reasoning in Ref-AVS through pre-decisional internal representations.

Thu 15 OctMultimedia
The gist
Too much thought can actually confuse computer systems when they try to find objects in videos by listening and looking, especially if the questions are simple. The authors found that systems can make mistakes when they think too long, so they built a way to predict when extra thinking will help or hurt. By checking a quick early guess, their method decides if more reasoning is needed and saves time and effort without losing much accuracy. This helps computers work faster and smarter by skipping unnecessary steps.
Open → 2610.21613v1

Multimodal learning improved by modeling joint higher-order interactions

HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling

Abstract: Self-supervised multimodal representation learning has achieved remarkable success across diverse domains, yet capturing synergistic information remains challenging due to the complexity of cross-modal interactions. Unlike the shared information across individual modalities, synergy arises when task-relevant signals emerge only from the joint configuration of multiple modalities and cannot be recovered from any modality in isolation. This work focuses on how to preserve the information capacity for such synergistic signals in multimodal representations. The key observation is that synergistic information is reflected in higher-order statistical dependence among modalities, which provides a principled target for explicitly modeling joint interactions. Motivated by this insight, we propose Higher-order Representation and Information Learning (HRIL), which constructs an empirical cross-moment tensor over modality embeddings to represent multi-way interactions. HRIL employs Tucker decomposition to obtain a core tensor, complemented by a synergy-aware regularizer that prevents energy concentration and preserves higher-order coupling capacity for synergistic information capture. Experiments on the controlled synergy task and real-world benchmarks demonstrate consistent improvements over existing multimodal contrastive methods, with notable gains on tasks dominated by synergistic interactions. Code is released at https://github.com/brightest66/HRIL.

Thu 8 OctArtificial IntelligenceMachine Learning
The gist
When machines learn from different types of information, like images and text together, it's hard to capture the unique insights that come only from mixing these sources. The authors found that these special combined signals show up in complex statistical patterns across the data types. They created a new method called HRIL that looks at these complex patterns using a special math tool called tensors, helping machines better understand how the pieces fit together. Tests show their method works better when it's important to learn from these combined signals.
Open → 2610.12393v1

Controlled fusion of audio video and text improves decision making with limited data

Controlled Acquisition and Abstention in Three-Channel Score Conflicts

Abstract: When audio, video, and text disagree, accuracy alone does not show whether to acquire another source or abstain. We study these choices in a controlled three-score benchmark: a policy observes two signed scores, may request the third at a cost, and can abstain. The primary reward is mechanism-specific: abstention is correct only for one designated ambiguity mechanism and is penalized under mixed corruption. Matched controls show that a threshold policy matches always-request decisions with fewer requests; its advantage over always-answer fusion depends on the reward assigned to that ambiguity. On a partially held-out synthetic split, the threshold policy reaches 0.789 +/- 0.006 targeted decision accuracy and 0.481 +/- 0.014 utility across 83 seeds. A three-score majority reference reaches 0.626 +/- 0.008 and 0.252 +/- 0.016, but uses more information. In a matched-budget test, a train-only value selector improves utility over no-query and matched-random policies at 10% and 25% budgets, while pair uncertainty has higher utility at every budget. At 50% and 63.7% budgets, the selector lowers utility despite slightly higher non-ambiguous accuracy. If all abstentions are scored incorrect, majority outranks the threshold policy in utility. At a central temporal setting, full-trace controls match the neural models while position perturbations separate them. On held-out-actor emotion clips, eight-frame fusion has opposite-signed accuracy differences for two encoder pairs, with both actor intervals containing zero; matched-request routing gains are small and uncertain. These results separate full-modality accuracy from pre-request selection value and show that selection value depends on budget and the observed-pair ranking.

Wed 7 OctMachine LearningMultimedia
The gist
When audio, video, and text information don’t agree, just measuring accuracy doesn’t tell us when to look for more information or decide to not answer. The authors created a setup where a system sees two scores, can pay to get the third, or choose to abstain from deciding. They found that a smart rule using thresholds picks when to get more info or abstain better than always answering or always asking for everything, depending on the situation. This helps show that choosing what data to use before asking for more is important and depends on how much data you want to spend and what’s already seen.
Open → 2610.10808v1

MUNITE enables flexible generation between any combination of modalities

MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation

Abstract: We introduce MUNITE, a latent-variable framework for flexible any-to-any multimodal generation that treats encoding and latent generation as the same inference problem under different amounts of observed evidence. Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation. Full observation recovers deterministic encoding, no observation recovers the latent marginal, and intermediate subsets define conditional latent inference, all within a single conditional flow model. A shared latent sample captures variation that must remain consistent across generated targets, while modality-specific generative decoders model the remaining uncertainty independently. To learn these conditional distributions from incomplete training examples, we extend conditional flow matching through self-distillation: predictions conditioned on richer available observations supervise the same model conditioned on smaller subsets at the same intermediate latent state. When the richer-evidence trajectory follows the exact conditional flow, this provides the same expected learning signal as full-target denoising. Across PolyMNIST-D-Q, FFHQ64, and image-text-audio, MUNITE achieves competitive or better generation quality and source-target alignment, with higher joint-generation coherence. In particular, it attains the highest coherence in all one-to-many and unconditional image-text-audio comparisons, showing the effectiveness of unified latent inference across diverse multimodal settings.

Wed 7 OctMachine Learning
The gist
Multimodal generation means creating outputs that mix different types of data like images, text, and audio. The new MUNITE method treats the process of understanding inputs and generating outputs as part of one system that works regardless of which data types are available. It uses a shared hidden space to keep consistent information across outputs, even if only some inputs are given. The authors tested MUNITE on various datasets and found it creates high-quality, coherent results across many tasks involving images, text, and sound.
Open → 2610.09866v1

LeAVJEPA simplifies audio visual learning with fewer components

LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

Abstract: Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.

Mon 5 OctMachine LearningComputer Vision and Pattern Recognition
The gist
Combining what we see and hear is tricky for computers, because they use lots of complex parts to learn together. The authors created LeAVJEPA, a simpler system that treats missing audio or video like a different view of the same moment, helping the computer learn how sounds and visuals match naturally. This method avoids common problems like losing meaningful information and works well on popular sound and video tests without complicated extra tools. Their approach shows that simpler setups can still understand audio-visual connections effectively.
Open → 2610.06226v1

Audio visual models reduce cross talk errors by focusing on relevant clues

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Abstract: Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions require different perceptual evidence, including audio, video, or their interaction. Notably, we observe that joint audio-visual inference can weaken the prediction even when a model can recover the correct answer from a single informative modality. For example, when asked which instrument is heard, a model may correctly predict violin from the audio alone. Once a video showing a guitar is added, its confidence in violin may drop. In this paper, we introduce Relevant Evidence Decoding (RED), a training-free method that identifies question-relevant evidence and selectively strengthens its contribution. RED uses pointwise mutual information to quantify the predictive support provided by audio and video beyond the question alone. It decomposes their joint contribution into audio, video, and residual interaction components. A question-only inference pass determines the required evidence type, after which the model augments the original audio-visual prediction with the corresponding PMI contribution. Across three audio-visual hallucination benchmarks and three AV-LLMs, RED improves accuracy over standard decoding by up to 7% on CMM, 6.3% on AVHBench, and 3.8% on SVHalluc, with an average relative time to first token of 1.5x standard decoding.

Fri 2 OctArtificial Intelligence
The gist
Audio-visual language models sometimes get confused when combining sound and video inputs, leading them to make mistakes called hallucinations. The authors found that trying to use both inputs together can weaken the model’s correct guess when one input alone was enough. They introduced a method called Relevant Evidence Decoding (RED) that figures out which clues (audio, video, or both) matter for each question and gives those clues more weight. This method improved accuracy when tested on multiple benchmarks without needing extra training.
Open → 2610.02976v1

Video generation models struggle to create varied motion and camera effects

DiVid: Diagnosing Dimension-Specific Diversity Collapse in Video Generation Models

Abstract: Despite remarkable progress, video generation models often produce highly similar outputs when repeatedly sampled from the same prompt, limiting their usefulness for creative exploration. Existing diversity evaluations primarily rely on global scalar metrics, which obscure where diversity collapses in the spatiotemporal space of videos. We introduce DiVid, a dimension-level diagnostic framework that decomposes video generation diversity into six interpretable dimensions: Semantic, Style, Subject, Scene, Motion, and Camera. Each dimension is measured through a reproducible computer-vision pipeline and analyzed alongside quality and instruction faithfulness to examine potential trade-offs. Systematic evaluation of representative video generation models reveals that diversity is highly dimension-specific: models with strong global diversity scores still collapse on specific factors, particularly Motion and Camera. These rankings persist after filtering unfaithful generations, indicating genuine capability differences rather than off-prompt outputs. Beyond measurement, controlled prompt interventions identify two fundamental bottlenecks: default mode convergence, where models fall back to dominant patterns under open-ended prompts; and realization gaps, where models fail to faithfully realize diverse, explicitly requested alternatives, particularly for temporal factors. The larger faithfulness losses for temporal factors highlight the difficulty of controlling motion and camera variation through text alone. DiVid thus shifts the study of diversity from measuring whether it exists to diagnosing where and why it collapses, and provides actionable directions for dimension-aware training objectives and control signals. The framework will be released to facilitate future research on diverse and controllable video generation.

Thu 1 OctComputer Vision and Pattern Recognition
The gist
Video generation models often produce very similar videos even when asked to create different ones from the same description. The authors developed DiVid, a new way to measure how diverse these videos are across six different parts, like motion or scene. They found that models might seem good overall but actually fail to vary certain parts, especially how things move or where the camera is. Their work helps explain why these models lack variety and suggests ways to improve them.
Open → 2610.01661v1

Audio and video combined improve reasoning in ai models

OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning

Abstract: Recent advances have enabled unified omni-modal models in understanding audio, vision, and language. However, existing benchmarks, training data, and learning methods largely treat the modalities independently, leaving the capability of audio-visual joint reasoning poorly evaluated and insufficiently elicited. We address this gap with a benchmark, data engine, and learning method. First, we introduce OmniReasoningBench, a benchmark where both audio and visual evidence are indispensable. It comprises 1,150 multiple-choice and open-ended questions across two tasks, reasoning over video and reasoning beyond video. Second, we develop a data engine OmniQA. It automatically constructs evidence-grounded QA pairs that explicitly necessitate audio-visual joint reasoning, together with time-stamped clue chains that guide the annotation of thinking process. Besides our benchmark, this engine produces training data OmniReasoning-SFT-112K and OmniReasoning-RL-19K. Finally, we propose an on-policy self-distillation method Modality-Factored Self-Distillation (MFSD). It evaluates each sampled response under modality-specific clue contexts, disentangling the contributions of individual clues and their cross-modal interactions for token-level credit assignment. With our training data and learning method, our model OmniReasoning-30B-A3B achieves 50.0% on OmniVideoBench and 42.5% on OmniReasoningBench, improving the base model Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively. Moreover, it delivers substantial gains on general and long-video benchmarks, including Video-MME-v2. We hope our work offers a solid step for facilitating future research in omni-modal joint reasoning.

Wed 30 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Many AI systems struggle to understand information from both sound and images at the same time. The authors created a new test and large datasets that specifically require AI to use audio and visual clues together to answer questions. They also designed a training method that helps models learn how to pay attention to both types of clues properly. Their improved AI model performs better on tests requiring this combined understanding than previous versions. This work helps move toward AI that can reason about what it sees and hears jointly.
Open → 2609.39490v1

OmniRoute improves efficiency in multimodal audio visual models

OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models

Abstract: Omnimodal large language models (Omni-LLMs) encode audio and visual streams into temporally interleaved token sequences for multimodal reasoning. However, processing long audio-visual token sequences incurs substantial prefill costs. Existing compression methods have made progress, but often overlook temporal changes in audio-visual semantic relevance. Motivated by temporal variation and local continuity, we propose OmniRoute, a training-free, two-stage compression framework. First, Temporal Evidence-Guided Budgeting (TEGB) derives chunk-wise modality preferences and initial leading-modality budgets from semantic relevance and local content variation. Second, Budget-Constrained Semantic Compression (BCSC) compresses the leading modality and then calibrates the follower's retention target using the actual retained fraction. For video, it combines spatiotemporal grouping with query-guided selection; for audio, it selects tokens based on encoder attention and query relevance, then merges residual tokens into context anchors under visual guidance. Experiments on four representative benchmarks demonstrate a better trade-off between inference efficiency and performance than competitive baselines. The code and interface will be released to facilitate further research.

Tue 29 SeptComputer Vision and Pattern Recognition
The gist
Processing both audio and video together in language models takes a lot of computing power, especially when handling long clips. The authors designed OmniRoute to reduce this workload by smartly choosing what parts of the audio and video to keep based on how important they are over time. It works in two steps: first, it figures out which modality (audio or video) matters more at different times, then it compresses that data accordingly while adjusting the other. This results in faster processing without losing much understanding.
Open → 2609.37052v1

Video sentiment analysis improves by splitting polarity and intensity tasks

Divide and Conquer: Mixture-of-Bottleneck Experts in Informative Ordinal Space for Video-based Multimodal Sentiment Analysis

Abstract: Video-based Multimodal sentiment analysis (MSA) must handle information from text, audio, and image sequence in human speaking videos, yet current methods often fail to integrate modalities with task awareness. Most models treat video sentiment prediction as a single task, overlooking its ordinal nature, and their fusion strategies struggle to capture diverse unique and synergic cues across modalities. To address these limitations, we adopt a divide-and-conquer perspective by reformulating MSA as an ordinal regression problem and decoupling it into polarity recognition and intensity prediction. Driven by information theory, we introduce a Mixture-of-Bottleneck (MoB) framework that assigns different latents to polarity- and intensity-specific experts for different modalities. With the learning of information bottleneck, each expert learns compact and task-relevant representations while filtering out redundancy and noise. A multimodal bottleneck routing fusion module then fuses these expert latents with hard mining strategy, guiding the prediction in the ordinal sentiment space. Extensive experiments on 4 MSA datasets and 4 language models show that MoB effectively leverages informative latents from diverse modalities and captures general sentiment structure. Beyond stronger performance, MoB comprehensively captures fine-grained intra- and inter-modal dynamics, enabling more trustworthy localization of nuanced video sentiment signals.

Wed 16 SeptMultimediaComputation and Language
The gist
Video sentiment analysis tries to understand feelings in videos using text, sound, and images together, but it is tricky because feelings have degrees and different parts work differently. The authors treat the problem as two tasks: recognizing positive or negative feelings and how strong they are. They introduce a method that uses small focused parts called bottlenecks to learn useful signals separately for these tasks from each kind of data. This helps the model better combine the different signals to predict feelings more accurately and understand subtle emotions in videos.
Open → 2609.18470v1