Papers for

multimedia software developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

OmniOPD improves joint audio and video generation stability

OmniOPD: Stable On-Policy Distillation for Joint Audio-Video Generation

Abstract: Reinforcement learning (RL) elevates joint audio-video generation, but optimizing heterogeneous, modality-specific audio and video rewards often introduces cross-modal interference. On-policy distillation (OPD) alleviates this issue by transferring the capabilities of specialized audio and video RL teachers into a single student model. However, a significant imbalance in cross-modal token counts leaves audio with sparse supervision, causing unstable distillation. We observe that naively increasing the audio loss weight fails to solve this problem since audio learning relies heavily on visual context, indicating richer visual cues are crucial to offset sparse audio signals. To address this, we propose OmniOPD, a modality-wise on-policy distillation framework designed for balanced and stable audio-video generation. First, we construct privileged visual information using asymmetric noise scheduling, allowing the audio teacher to generate more informative targets from cleaner video context. Second, we introduce grouped multi-noise sampling to expand audio supervision coverage and stabilize training. Third, we introduce timestep-wise importance weighting based on teacher-student transition divergence, preserving reliable supervision while down-weighting unreliable learning signals, to stabilize fluctuations in audio learning. Evaluated on JavisBench and VBench demonstrate OmniOPD preserves audio quality while significantly enhancing video generation, achieving effective joint distillation.

Mon 2 NovComputer Vision and Pattern RecognitionArtificial IntelligenceSound
The gist
Producing videos with matching sound is tricky because the computer must learn from both visuals and audio, which can interfere with each other. The authors found that teaching the computer using separate specialized models for audio and video helps, but audio signals often get weaker during this combined learning. To fix this, they developed OmniOPD, which uses cleaner video information and smarter sampling techniques so the audio model learns better alongside video. This results in more stable and balanced creations of videos with good audio and visuals together.
Open → 2611.02803v1

Compact audio visual agent cuts toolset and costs by 70 percent

Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

Abstract: Fine-grained audio-visual question answering (AVQA) requires models to resolve complex cross-modal correspondences and temporally grounding from sparsely distributed evidence. Omni-modal agents, which autonomously orchestrate perception tools to actively collect evidence, achieve strong performance, but it remains unclear whether active perception from both audio and visual modalities depends on broad specialized toolsets, especially audio or visual modality-specific tools. Such systems may also introduce long execution trajectories and substantial inference cost. We investigate whether comparable performance can be retained with a compact workflow exposing only essential perceptual capabilities. Starting from the specialized-tool OmniAgent, we conduct staged controlled experiments that prune the toolset and progressively consolidate modality-specific tools into omni-modal tools, motivating a two-tool agent comprising only Omni-modal captioning and Omni-modal QA. Across three widely used AVQA benchmarks, our compact agent achieves comparable or better accuracy than the specialized-tool agent while reducing inference cost by around 70%.

Wed 28 OctArtificial IntelligenceComputation and Language
The gist
Answering questions about events or scenes that combine sound and visuals can be tricky because it requires understanding details across both senses. The authors studied a system that uses many specialized tools for audio and visual analysis to answer these questions, but such systems can be slow and costly. They found that by carefully reducing these tools to just two general-purpose ones that handle both senses, the system keeps or even improves accuracy while running much faster. This means simpler, cheaper systems can still do complex audio-visual reasoning well.
Open → 2610.38960v1

Code based video agents rated across diverse real world tasks

VibeVideoBench:Evaluating Multimodal Agents on Code-Driven Video Generation

Abstract: Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a ruleconsistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, judged by three VLM judge panels and validated with blind human preferences. Across nine frontier agent systems and 2,151 scored rollouts, the benchmark is far from solved: GPT-6 Astra and Claude Opus 5.5 lead clearly, yet even the best system fails a core requirement on about a quarter of tasks. Whether a video arrives at all matters as much as its quality, and three quarters of the lost videos are final renders started too late, run too slowly or never started. The judges agree with each other and with human preferences on system rankings, but only moderately on individual videos.

Mon 26 OctComputer Vision and Pattern Recognition
The gist
Video generation models often cannot create specific and detailed videos like exact formulas or synchronized narration. The authors present VibeVideoBench, a new test set with 239 video tasks in English and Chinese, covering topics like lessons, data stories, and music videos. They tested nine top AI agents that write code to generate videos with sound. Even the best agents fail about a quarter of the time, often because they run too slowly or don’t finish rendering. The evaluation combines automated checks with human-like judge panels to rate quality and task success.
Open → 2610.35864v1

Audio video generation speeds up with selective caching approach

CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

Abstract: Diffusion-based audio-visual generation methods typically perform joint denoising of audio and video latents throughout the entire sampling trajectory, repeatedly executing modality-specific denoising branches and cross-modal attention at every timestep. However, we observe that such uniform joint computation is not always necessary. The key is a modality-asymmetric recoverability: audio-side drift introduced by mid-stage caching can be substantially reduced by only a few terminal joint denoising steps, whereas video-side degradation caused by caching leads to persistent loss of motion and temporal details. This asymmetry suggests that inference acceleration should selectively cache recoverable audio-side computation while preserving full video refinement. Motivated by this observation, we propose \textbf{Convergence-Adaptive Audio-Visual Cache (CAVC)}, a training-free modality-asymmetric caching framework. CAVC adopts a sandwich-structured computation allocation: after detecting that audio denoiser outputs enter a cache-tolerant regime, it caches and reuses the audio denoiser output to bypass audio denoising and cross-modal attention in the middle stage, while the video branch continues independent spatio-temporal refinement; a few joint steps are then restored at the end to reduce accumulated cross-modal drift and improve synchronization. Experiments across multiple public benchmarks demonstrate that CAVC achieves 1.10--1.35x wall-clock acceleration, improves several visual dynamics and temporal statistics, and maintains competitive audio fidelity and audio-visual alignment.

Thu 22 OctComputer Vision and Pattern RecognitionSound
The gist
Generating synchronized audio and video through diffusion models usually requires equal work on both to maintain quality, which can be slow. The authors noticed audio can be cached and reused in the middle of generation without hurting much, while video needs more careful, continuous processing. They created a caching method called CAVC that smartly saves audio computations mid-way and finishes with some joint cleanup, speeding up generation while keeping good video and audio quality. Their tests showed faster results with better video motion and timing, and music sounds just as good as before.
Open → 2610.30854v1

Audio visual benchmark tests evidence based reasoning in omni language models

EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Abstract: Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning. Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer. In addition, textual or unimodal shortcuts may allow models to answer correctly without analyzing the necessary audio-visual evidence, further weakening the reliability of evaluation benchmarks. To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning. We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts. Beyond high-quality question-answer pairs, we require models to simultaneously return the answer and supporting evidence intervals in order to evaluate the complete reasoning chain. We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness. Across 7 open-weight and 17 proprietary Omni LLMs, Gemini3.7-Flash achieved the highest overall accuracy (79.4%), C.G. (70.8%), and B.G. (35.1%). At Level 3, it reached 79.1% accuracy, but its B.G. was only 12.3%. These results show that even the most advanced models still have room for development in grounded audio-visual reasoning.

Tue 20 OctComputer Vision and Pattern Recognition
The gist
It’s hard to know if computer programs that answer questions about videos and sounds really understand what they see and hear or are just guessing. The authors created EvaOmni, a careful test containing videos with questions that need finding evidence in both sound and pictures to answer correctly. They made sure the questions and answers rely fully on audio-visual clues and that models must show the exact proof behind their answers. They also introduced scores to check how well models find and pinpoint that proof. Their tests show even the best models still struggle to fully connect their answers to the right evidence.
Open → 2610.28444v1

Agentic AI enables secure secret message hiding in videos

SemVideoSte: Agentic AI for Cross-Modal Semantic Video Steganography Communication

Abstract: We propose an agentic artificial intelligence (AI)-driven cross-modal semantic video steganography communication (SemVideoSte) scheme for secure cross-modal information transmission. Agentic AI interprets the service context to determine a semantically matched public prompt and coordinates video generation, message embedding, semantic transmission, and message recovery process. A multi-mask mechanism combines variational autoencoder (VAE) reconstruction robustness, semantic transmission robustness, and latent separability to select reliable embedding positions. Meanwhile, a private digital token controls the random seed and guidance offset to construct positive and negative terminal latent states, enabling coverless message embedding during the final diffusion sampling step. Simulation results on open-source datasets demonstrate that SemVideoSte achieves high embedding capacity, reliable message recovery, and strong semantic consistency under different channel conditions. In addition, eavesdroppers without the correct digital token cannot reliably recover the secret message, and the generated stego videos effectively resist the steganalysis detection. We further establish a prototype testbed to verify the practical feasibility of SemVideoSte.

Sun 18 OctCryptography and Security
The gist
Sending secret messages hidden inside videos can be tricky because the message needs to stay safe and not be easily found. The authors created a smart AI system that understands the situation and carefully hides secret messages in videos without altering them much. This makes sure the messages are safe from people trying to find or steal them. Their method works well on test data and even in real-world setups, making secret video messaging more secure and reliable.
Open → 2610.25257v1

Intern Lumina U2 improves visual understanding and image generation with multi-codebook tokens

Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation

Abstract: Unified multimodal generation covers discrete, continuous, and hybrid paradigms. Discrete approaches with single-codebook visual tokenizers align naturally with text modeling but have limited representational capacity. Continuous and hybrid approaches retain richer visual representations, but they model text and images in heterogeneous spaces with different training objectives, potentially introducing optimization conflicts between modalities with misaligned representation spaces. These choices expose a central question: how to efficiently model high-capacity visual representations alongside textual content in a shared optimization space within a single backbone. We present Intern Lumina U2, to our knowledge the first diffusion language model to represent multi-codebook visual tokens in a fully discrete space. Each visual position is represented by a group of codes from complementary codebooks, increasing its representational capacity while preserving compatibility with discrete text modeling. At the input, the embeddings are concatenated at each position; at the output, a lightweight multi-codebook autoregressive head predicts the codes within each position sequentially, while the diffusion backbone processes all positions in parallel. Built on the unified 4D representation of AToken, Intern Lumina U2 supports image, video, and 3D inputs within a single backbone and achieves leading performance among unified models in visual understanding, image generation, and image editing. We further build a full-stack training, inference, and evaluation system on both Ascend and NVIDIA platforms to verify the cross-platform effectiveness of our model. We also improve the inference efficiency through a stability-aware sampling strategy, which combines confidence-plateau-guided unmasking approach with drift-triggered cache refreshing mechanisms, accelerating image generation by 4.45x.

Wed 14 OctComputer Vision and Pattern Recognition
The gist
Combining text and visual information in a single model is challenging because images and words are very different types of data. The authors present Intern Lumina U2, which uses multiple sets of visual codes to better represent images while still working smoothly with text. This helps the model understand and create images, videos, and 3D content more effectively. They also made the system faster and compatible with different computing platforms.
Open → 2610.20818v1

EventLens improves video understanding by learning event structure

EventLens: Event-Structure Reinforcement Learning for Video Understanding

Abstract: Recent multimodal large language models (MLLMs) achieve strong results on video question answering, yet often fail to recover the temporal structure that makes a video coherent. They may merge neighboring events, miss single-event progression, or infer implausible relations across events. We study structured temporal understanding as the ability to recover event structure from visual evidence, including event boundaries, single-event progression, and multi-event relations. We argue that this gap arises from a mismatch between current post-training objectives and temporal reasoning, where caption-style supervision allows models to rely on language shortcuts rather than visual temporal grounding. To address this, we propose EventLens, a vision-centric reinforcement learning framework that learns video understanding through event-structure recovery. Instead of reconstructing captions, EventLens trains models to recover temporal structure under controlled perturbations, such as altered temporal granularity, reversed progression, and shuffled event order. We instantiate this framework with a three-level temporal ontology and derive three verifiable task families: event segmentation, progression discrimination, and multi-event relation reasoning. These tasks admit deterministic rewards, enabling task-specialized GRPO training without human preference labels or learned reward models. We further introduce multi-teacher on-policy distillation (MT-OPD) to consolidate specialized policies into a unified model. Experiments on Qwen3-VL backbones show consistent improvements on temporal grounding and event-sensitive reasoning benchmarks, while preserving general video-QA performance. Learning curves further demonstrate that EventLens achieves comparable downstream transfer with significantly reduced training cost compared to direct mixed RL optimization.

Mon 12 OctComputer Vision and Pattern RecognitionArtificial IntelligenceMultimedia
The gist
Videos are made up of events that happen in order, but current AI models often struggle to tell the difference between these events or understand their timeline correctly. The authors studied this problem and found that usual training methods let models rely too much on language clues instead of really understanding the video timing. They created EventLens, a new way to teach AI that focuses on recognizing the order and boundaries of events by making it solve puzzles about how events are arranged in time. This method helps AI better understand videos while keeping its ability to answer questions about them.
Open → 2610.16172v1

Multimodal models learn to adapt thinking style and length for tasks

Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning

Abstract: The introduction of long chain-of-thought (CoT) reasoning and external tools has substantially improved the performance of multimodal large language models (MLLMs) on general and reasoning tasks. However, existing fast-slow reasoning paradigms mainly regulate textual reasoning length, while visual computation introduced by external tools or visual tokens is often treated as an auxiliary component. This separation prevents MLLMs from jointly optimizing visual perception and textual reasoning. To address this issue, we propose L2T (adaptive Look, flexible Thought), a unified reasoning framework that jointly optimizes the form and depth of computation. L2T first discretizes tool-produced visual states into visual tokens, bringing visual perception and textual reasoning into a shared autoregressive space. Based on this unified representation, L2T introduces a modality-aware routing reward to select appropriate reasoning modes and a difficulty-aware fast-slow strategy to adapt reasoning depth according to task complexity. Experiments on general perception and reasoning-intensive tasks demonstrate the effectiveness of L2T. Compared with explicit tool-calling approaches, L2T maintains competitive performance while achieving 1.7--3.6 times higher throughput.

Sun 11 OctComputer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
The gist
Multimodal large language models, which understand both pictures and text, usually handle visual and textual reasoning separately. This separation limits how well they can work together on complex tasks. The authors created a method called L2T that treats visual and text reasoning as one process, deciding how much and in what way to think based on the task. Tests show L2T performs as well as current methods but can work 1.7 to 3.6 times faster.
Open → 2610.15810v1

Text to video models vary in expressing and controlling emotions accurately

EmoTrace-Bench: Diagnosing Emotional Expression in Text-to-Video Generation through Controlled Cues and Intensity

Abstract: Text-to-video (T2V) models are increasingly used to generate videos with targeted emotional expressions, yet existing evaluation methods mainly rely on holistic video-text alignment and provide limited insight into whether failures arise from target-emotion expression, intensity control, or specific affective cues. We introduce EmoTrace-Bench, a fine-grained diagnostic benchmark that controls emotion intensity and selected affective cues in prompts and evaluates generated videos at multiple levels. EmoTrace-Bench contains 448 controlled prompts covering eight emotions, two intensity levels, and diverse cue conditions. Using 12 T2V models, we generate 5,376 videos and evaluate target-emotion presence, perceived intensity, and ten audiovisual dimensions of emotional expression with an automated evaluator calibrated and validated against human annotations on a subset of the data. Results reveal substantial differences across models and prompt designs. Prompts that explicitly describe a character's emotional reaction are followed more reliably than prompts that describe only events or scenes, whereas adding more affective cues or using more explicit cinematic audiovisual language does not consistently improve emotional generation. These results illustrate how aggregate alignment scores may overlook localized differences in target-emotion expression, intensity control, and cue realization. EmoTrace-Bench provides a traceable and interpretable framework for diagnosing emotional video generation without modifying existing generation pipelines, and we release the prompts and the automated evaluation rules.

Sun 11 OctComputer Vision and Pattern Recognition
The gist
Generating videos from text that show specific emotions is tricky. The authors created a special test called EmoTrace-Bench to check how well these models show the right emotions, how strong those emotions appear, and which details in the text help. They used many prompts and videos across different emotions and intensity levels, finding that some ways of describing emotions work better than others. This test helps understand exactly where these video generation models do well or poorly without needing to change the models themselves.
Open → 2610.15452v1

Audio impact on visual prediction clarified by new four gate checks

CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks

Abstract: Removing audio can shift a visual predictor's output without showing whether the task uses the intended audio information. We introduce Conditional Input and Visual Interpretation Checks (CIViC), a four-gate protocol that links an interpretation to an intervention, held-out task comparison, or endpoint control. We demonstrate CIViC on actor-disjoint face-raster tests, a synthetic audio-feature mixture, and source-video-disjoint SpeakerVid proxy tasks. In the static face-raster test, cross-actor audio replacement meets the registered detection rule with a small effect; on a separate temporal endpoint, constant-audio fits score better than audio-trained fits in four matched seeds. In the synthetic mixture, the condition-augmented fit has higher 1,000-step RMSE than the audio-only fit (0.1807 vs. 0.1707), with the gap shrinking during training. In the SpeakerVid proxy, the five-feature-zero fit (five low-level audio summaries zeroed; identity, dialogue, and condition inputs retained) has lower whole-frame grayscale RMSE than aligned audio across four held-out conditions and five paired seeds, while aligned audio has lower segment-drift error. The results show that output activation, held-out error, and endpoint controls support different interpretations across these proxy tasks.

Sun 11 OctMachine LearningComputer Vision and Pattern RecognitionSound
The gist
It can be hard to tell if sounds actually help a computer guess what it sees in videos because changing audio might not always mean the machine uses it. The authors created a new method called CIViC that uses four test points to better check if audio is truly involved in the predictions. They tried their method on fake and real video tasks showing that different checks give different clues about how audio influences the visual guess. This makes the analysis of audio-visual tasks more reliable and clear.
Open → 2610.15327v1

Benchmark reveals strengths and weaknesses of audio visual captioning models

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

Abstract: Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.

Thu 8 OctComputer Vision and Pattern Recognition
The gist
Understanding how well AI models can describe what they see and hear together is hard because current tests either check the whole description at once or look only at parts. The authors created OmniCapBench, a new way to evaluate AI that breaks down descriptions into small, clear parts to check accuracy more precisely. They tested many videos and found that while some AI models can recognize details well, they struggle with keeping track of identities over time and linking sounds with images correctly. This helps show where future AI improvements should focus.
Open → 2610.12458v1

Syn omni improves cross modal embeddings with structured specialization

Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings

Abstract: Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.

Thu 8 OctComputer Vision and Pattern RecognitionArtificial IntelligenceInformation Retrieval
The gist
Combining data from images, videos, and sounds into one smart system is tricky because each type of data is different, but they also share some similarities. The authors designed Syn-Omni, a new way to handle this by keeping specific details for each data type separate while still letting them work together. They developed techniques to first focus on individual data types and then gradually mix them for better understanding. This approach was tested on many tasks involving pictures, videos, and audio, consistently doing better than previous methods.
Open → 2610.12256v1

Benchmark evaluates prompt enhancers for video generation tasks

Towards Unified Evaluation of Prompt Enhancers for Video Generation

Abstract: Modern video generators can realize increasingly complex visual narratives, positioning the prompt enhancer (PE) as a critical bridge from concise user instructions and multimodal references to structured cinematic plans. However, existing PE evaluation relies on rendered videos, imposing substantial computational and human costs, slowing PE training and iteration, and conflating PE quality with downstream generator behavior. To address this gap, we introduce PEBench, the first unified benchmark for direct PE evaluation across text-to-video, image-to-video, and reference-to-video prompt enhancement. It comprises 1,100 expert-verified cases and 1,005 visual assets, spanning 35 fine-grained tasks with diverse temporal, cinematic, audiovisual, and multi-reference requirements. In addition, we develop PEBench evaluation, an evidence-grounded framework that combines modality-aware fact extraction with rubric-based assessment across 24 criteria. Our systematic evaluation of representative open- and closed-source PE methods reveals an emerging shift from fine-grained descriptive expansion toward intent-preserving cinematic planning, while the caption-reconstruction and forward-refinement methods show complementary strengths in cinematic coverage and semantic fidelity or internal coherence, respectively. Human validation shows that PEBench scores align closely with expert judgments of enhanced prompts and downstream videos from Wan3.0 and MiniMax-H3, indicating that prompt-level evaluation reliably reflects downstream utility.

Thu 8 OctComputer Vision and Pattern Recognition
The gist
Creating videos from simple instructions can be tricky because the instructions need to be detailed and clear for the computer to understand. The authors provide a way to test how well tools called prompt enhancers improve these instructions before making the video. Their new benchmark, PEBench, lets developers quickly check the quality of these enhanced instructions without generating full videos every time. This helps speed up improving video creation by focusing on better planning rather than just the final video output.
Open → 2610.11736v1

Models trained on different data can align without paired examples

Shared Geometry As A Rosetta Stone: Cross-Modal Alignment Without Paired Data

Abstract: Multimodal representations enable zero-shot classification and retrieval, but aligning independently trained models usually requires large amounts of paired data. Yet, the Platonic Representation Hypothesis suggests that models trained on different modalities may converge spontaneously toward a shared representation geometry. But then, do we even need paired examples for cross-modal alignment? Remarkably, we show that paired examples are unnecessary for coarse cross-modal alignment. Our simple Wasserstein Procrustes method with a coarse geometric initialization aligns two disjoint embedding sets by estimating a single orthogonal map without seeing any pairs. Across datasets, modalities, and unimodal models, we show that we can consistently align independently trained representations without pairs, and standard geometric alignment metrics accurately predict when this is possible. Nevertheless, we can naturally benefit from paired examples. In the very few-pair regime, our method substantially outperforms existing ones, while staying competitive with pair-based methods with more added examples. Finally, we demonstrate that the resulting alignments can enable text-to-image generation without paired examples. These results show that independently trained models often share enough geometry to establish cross-modal correspondence with little or no paired data.

Wed 7 OctMachine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Models that understand different kinds of data, like images and text, usually need examples that match across these types to work together well. The authors found that models trained separately can still align their understanding using shared geometric patterns without needing any paired examples. They developed a straightforward method that matches these models by looking at their overall shape instead of pairs. This method helps improve tasks like generating images from text, even when matching examples are scarce or missing.
Open → 2610.09411v1

Synthetic framework improves reasoning across multiple videos

SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation

Abstract: Reasoning across videos requires aligning events, matching identities, comparing motion, and integrating partial observations. Evaluating these capabilities and testing how to improve them requires both reliable labels and targeted supervision. We introduce SYNCR, a simulator-grounded framework that connects these two needs through shared task generators. Built on Habitat, Kubric, and CLEVRER, SYNCR derives answers from environment state and provides 4,000 evaluation questions and 15,960 training questions over disjoint videos, spanning eight cross-video reasoning tasks. Visual ablations and human evaluation assess dependence on the supplied evidence and answer recoverability. Evaluation of 22 multimodal large language models reveals persistent difficulties in physical comparison and scene integration that increasing model size does not consistently resolve. Supervised fine-tuning raises Qwen3-VL-8B's average SYNCR accuracy from 32.6% to 61.6%, with gains extending to task configurations and video sources absent from training for those tasks. Transfer to real footage is most consistent for temporal ordering: accuracy improves by 9.0-20.5 percentage points on constructed Assembly101 and Panoptic ordering sets across three checkpoints spanning two model families and two model sizes, with additional gains on existing temporal reasoning benchmarks. These results establish SYNCR as a controlled setting for diagnosing cross-video reasoning failures, testing their learnability, and identifying where synthetic supervision transfers.

Tue 29 SeptComputer Vision and Pattern RecognitionMachine Learning
The gist
Figuring out connections and comparisons across different videos is tough for AI because it needs to match events, recognize the same people or objects, and integrate incomplete information. The authors created SYNCR, a testing system with simulated videos and thousands of questions to evaluate these skills in AI models. They found many large AI models struggle with comparing physical traits and understanding whole scenes, even if the models are larger. Fine-tuning a strong model with SYNCR questions greatly improved accuracy, and some improvements even carried over when tested on real-world video tasks.
Open → 2609.37918v1

Codebook method helps transfer knowledge between different data types

Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features

Abstract: Cross-modal knowledge distillation transfers knowledge from a teacher modality to a student modality. Existing feature-level alignment methods typically assume that teacher and student features reside in structurally alignable representation spaces. However, this assumption does not hold when cross-modal features are structurally heterogeneous and lack clear unit-level correspondence, such as 2D spatial visual grids and 1D temporal audio sequences, thereby limiting the applicability of feature-level alignment. To address this challenge, we propose a cross-modal distillation framework that enables effective knowledge transfer across structurally heterogeneous feature spaces via a vector-quantized codebook. Specifically, teacher features are abstracted into a set of vector-form codes regardless of their original feature structure, and the selected codes serve as concept-level anchors for student learning. Code selection is guided by both task relevance and student compatibility, allowing the student to receive transferable teacher knowledge without requiring direct unit-level feature alignment. Experimental results across diverse cross-modal distillation scenarios demonstrate the effectiveness of the proposed framework on classification and semantic segmentation tasks.

Tue 29 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Sometimes computers learn better by copying what another computer 'knows,' but this gets tricky when the two computers look at very different kinds of information, like pictures versus sounds. The authors created a way to turn complex information into a common set of simple codes to help the student computer learn from the teacher computer without matching every detail directly. This method helps transfer knowledge even when the data types are very different in shape or format. The authors tested this on tasks like recognizing objects and segmenting images and found it works well.
Open → 2609.37243v1

SyncRA improves timing connections between sounds and images in videos

SyncRA: Learning Temporal Correspondence in Omni-Modal Models

Abstract: Recent omni-modal models demonstrate strong perception of audio and visual inputs, yet often struggle to connect what they hear with what they see at the same moment. This weakness in temporal correspondence can cause models to associate spoken cues with the wrong visual scenes, producing plausible answers grounded in incorrect audio-visual pairings. We diagnose this problem through controlled temporal swaps, revealing that model answers do not reliably follow changes in these pairings. To address it, we propose Synchrony-Guided Representation Alignment (SyncRA), a lightweight method for strengthening temporal correspondence between audio and vision. Specifically, SyncRA contrasts intermediate audio-visual representations within each video, aligning matching moments while separating mismatched ones to capture local temporal correspondence within a shared global context. The objective derives supervision directly from existing input timing, requiring no additional annotations and leaving inference unchanged. We evaluate SyncRA across four open omni-modal models spanning different sizes and architectures on five public video benchmarks. SyncRA consistently outperforms answer-only fine-tuning across all model-benchmark combinations, while substantially improving the ability to track changing audio-visual pairings in controlled evaluations. These results demonstrate that lightweight, targeted supervision can effectively strengthen temporal correspondence and translate into broad improvements in audio-visual question answering.

Mon 28 SeptComputer Vision and Pattern RecognitionArtificial IntelligenceMultimedia
The gist
Many AI models that look at videos and listen to sounds have trouble matching the right sound to the right image at the same time. This can cause them to misunderstand what they see and hear together. The authors found that these mistakes happen because the models do not reliably link sound and visuals by their timing. They created SyncRA, a method that teaches models to better connect sounds and images happening together without needing extra labels or changing how the models work when watching videos. This method improved several popular AI models on tests where they had to answer questions about videos with both sound and pictures.
Open → 2609.34363v1

Multimodal transformers build virtual encoders inside their layers

Virtual Encoders in Multimodal Transformers

Abstract: Multimodal language models traditionally rely on dedicated perceptual encoders to construct task-usable representations. More integrated architectures have recently emerged, which instead expose the shared transformer to lightly projected patches, audio frames, or discrete visual tokens. Where does this encoding happen when such representations are not provided? We find that the transformer can internalize this missing computation, constructing task-usable perceptual representations within its own early-to-middle layers before the downstream language model. We call this computational structure a Virtual Encoder. Across linear probing, similarities to perceptual encoders, and causal analyses, we identify signatures of this structure in models that receive perceptual tokens without continuous encoder-derived features. These analyses also suggest that the boundary between perception and language processing need not coincide within an architectural module. Instead, encoder-like computation can emerge as a functional regime within a shared transformer, providing a new perspective for understanding where and how multimodal models process perception.

Tue 22 SeptComputer Vision and Pattern Recognition
The gist
Multimodal language models usually use separate parts to understand images, sounds, or other sensory data before using language. The authors found that some advanced models can do this sensory processing inside their own transformer layers without separate encoders. These internal computations, called Virtual Encoders, happen early inside the model and make the sensory data ready for language tasks. This changes how we think about where perception and language processing happen in multimodal AI.
Open → 2609.26513v1

Multimodal tokens unify images and text for better retrieval and generation

FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

Abstract: Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream generative model. This setup bottlenecks generative performance behind frozen embeddings. To bridge this gap, we revisit joint multimodal representation learning and generation to produce linearly interpolatable embeddings that are directly consumable by generative decoders. We present FLAT (Flexible-Length Aligned Transmodal representations), a representation pre-training framework that jointly optimizes a shared multimodal encoder alongside downstream text-to-image (T2I) and image-to-text (I2T) decoders. By combining contrastive alignment with bidirectional cross-modal generative objectives, FLAT ensures its representations function as both discriminative semantic descriptors and generative conditions. Architecturally, FLAT maps visual and textual inputs into a unified continuous 1D sequence space, applying nested dropout over prefix-K tokens to enable dynamic output lengths. A single pre-training stage allows FLAT to perform cross-modal retrieval and generation across variable prefix K, achieving a T2I GenEval score of 71.1. Task-specific fine-tuning aligns model performance with state-of-the-art baselines: 83.1 GenEval on T2I generation; 40.5 BLEU-4 and 138.6 CIDEr on MS-COCO image captioning; and Recall@5 scores of 86.8 (I2T) / 75.8 (T2I) on MS-COCO alongside 98.3 (I2T) / 93.6 (T2I) on Flickr30K. Finally, qualitative evaluations demonstrate that FLAT representations natively support linear interpolation, latent space arithmetic, and zero-shot composed retrieval.

Tue 15 SeptComputer Vision and Pattern Recognition
The gist
Combining images and text for tasks like captioning or image generation is hard because the two are usually processed separately. The authors developed a way to turn both images and text into a shared 1D sequence of tokens that work well for both recognizing and creating content. This helps a single model handle searching for images from text, generating images from text, and describing images with text, all with a flexible number of tokens. Their approach improves the quality of these tasks and supports interesting features like blending two images or texts smoothly.
Open → 2609.16591v1

Real-time video compression using deformable 2D Gaussian splatting

Deformable 2D Gaussian Splatting for Efficient 4K Video Compression

Abstract: Ultra-High-Definition (UHD) video presents significant challenges for efficient storage and real-time decoding. Learning-based methods, such as Neural Video Compression (NVC) and Implicit Neural Representations (INR), achieve competitive rate-distortion performance but suffer from high decoding latency and excessive memory usage. Meanwhile, Gaussian Splatting has recently attracted attention in the computer graphics community due to its ultra-fast rendering and high-fidelity visual quality. Despite these advantages, its application in video compression remains largely unexplored. To bridge this gap, we propose a real-time video compression framework that represents and compresses a Group of Pictures (GOP) using a coarse-to-fine multi-scale 2D Gaussian Splatting (2DGS) structure coupled with a lightweight deformation network. Experiments demonstrate that our method delivers rate-distortion performance in LPIPS that surpasses H.265 and other state-of-the-art learning-based video compression methods. Our work demonstrates the potential of Gaussian Splatting as a practical solution for efficient high-resolution video compression.

Sat 12 SeptComputer Vision and Pattern Recognition
The gist
Ultra-high-definition videos require a lot of storage and are slow to decode in real time. The authors propose a new way to compress videos by using a 2D Gaussian Splatting method combined with a lightweight deformation network. This approach can compress groups of video frames quickly and efficiently, resulting in better video quality compared to the popular H.265 standard and other recent AI-based methods. Their method is especially suited for very high-resolution videos, like 4K.
Open → 2609.14129v1

Multimodal emotion recognition improves with adaptive fusion and facial geometry

Multimodal Emotion Recognition in Conversations via Class-Wise Adaptive Modality Fusion and Affective Geometry

Abstract: Emotion Recognition in Conversations (ERC) requires integrating heterogeneous textual, audio, and visual cues while accounting for conversational context and emotional dynamics. We extend the Self-Distillation Transformer architecture for ERC with appearance+geometry visual representations, class-wise adaptive modality fusion, and a valence-arousal prior for affective transitions. On the MELD and IEMOCAP datasets, geometry-enhanced visual representations improve weighted F1 by 0.27 and 4.36 points over appearance-only features, respectively, while class-wise adaptive fusion provides further gains of 0.17 and 0.25 points over the original softmax gate. The valence-arousal prior yields targeted improvements of 0.30 and 0.74 accuracy points on emotionally shifted utterances while preserving performance on stable turns. These results indicate that structured facial cues, emotion-dependent modality weighting, and affective geometry provide complementary benefits for multimodal ERC.

Wed 9 SeptComputer Vision and Pattern Recognition
The gist
Understanding emotions in conversations is hard because people express feelings through words, tone, and facial expressions. The authors improve emotion recognition by combining facial appearance and precise facial movements, adjusting how much each type of signal counts depending on the emotion, and using a model of how emotions change over time. They tested their approach on two conversation datasets and found better accuracy, especially when emotions shift during the conversation. This shows that using detailed facial cues and emotion-aware mixing of signals helps machines understand feelings more reliably.
Open → 2609.09924v1