Papers for

multimedia content creators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Reinforcement learning improves multi image editing consistency

DeCoRL: Decomposed Consistency Reinforcement Learning for Multi-Image Composition

Abstract: Multi-image composition (MICo) is a critical yet challenging capability for large-scale image editing models, where consistency degrades rapidly as the number of reference images increases. While supervised fine-tuning (SFT) can equip models with basic MICo ability, it provides only implicit supervision on the composed image and often fails to learn reliable object-level correspondence across multiple references. We propose DeCoRL, a reinforcement-learning (RL) framework that improves MICo consistency by decomposing holistic MICo consistency into object-level consistency. DeCoRL first establishes instruction-relevant object correspondences across reference images and localizes the corresponding regions in the generated image. It then uses a VLM-based reward model trained on single-object consistency annotations to produce fine-grained, object-wise feedback, which is aggregated into a multi-object consistency reward to guide RL post-training. We further introduce DeCo-Bench, a benchmark covering compositions from 2 to 6 reference images. Experiments on DeCo-Bench and the public MICo-Bench demonstrate consistent improvements in MICo consistency.

Thu 29 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Editing images using several reference pictures at once is hard because keeping everything consistent across images is tricky. The paper presents a new approach called DeCoRL that teaches a model to focus on each object separately when combining images, so it keeps the details right. It uses a special reward system that checks object-level accuracy and helps the model learn better. They also created a new benchmark to test how well models can combine from two to six images, showing improved results.
Open → 2610.39265v1

Graph method improves emotion recognition in artworks over vision language models

Beyond Vision-Language Models: Visual-Semantic Affect Graphs for Emotion Recognition in Artworks

Abstract: Emotion recognition in artworks remains challenging because affective meaning is often subjective, implicit, and shaped by relationships between visual form and semantic associations. While vision-language models provide strong multimodal representations, direct image-to-emotion prediction may overlook explicit relations among formal, semantic, and affective cues that support emotional interpretation. This paper proposes Visual-Semantic Affect Graphs (VSAG), a graph-based framework that represents artworks through interpretable feature nodes, including color, brushstroke, composition, light-shadow, line quality, and extracted visual entities. The graph learns feature-emotion associations for dominant emotion prediction, with valence-arousal structure modeled through an optional hierarchical auxiliary affect module. We evaluate VSAG and VSAG-TopK against direct VLM inference using Qwen3-VL and InternVL3.5 on EmoArt-5k. Results show that direct VLMs provide useful but modest baselines, while graph-based models improve dominant emotion prediction. In particular, the hierarchical valence-arousal variant of VSAG achieves the best overall performance, improving accuracy, macro-F1, and weighted-F1 over both direct VLM inference and the emotion-only VSAG baseline. Per-class results indicate that graph-based models and VLMs capture partly complementary signals, while valence-arousal structure is most useful when integrated as a coarse-to-fine affect hierarchy rather than as a simple post-hoc correction. These findings suggest that explicit visual-semantic structure can support both prediction and interpretation in artwork emotion recognition.

Mon 26 OctComputer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
The gist
Understanding emotions in artworks is hard because feelings can be very personal and subtle. The authors created a new method that breaks down artworks into parts like colors, brushstrokes, and shapes, then connects these parts in a graph to predict the emotions the art expresses. This approach did better than using large vision-language models alone, especially when emotions were organized by general feelings like positivity or energy. Their work shows that explicitly looking at how different art features relate helps computers better guess the emotions in artwork.
Open → 2610.35244v1

Direct forcing improves few-step streaming text to video generation

Direct Forcing: A Single-Stage Training Recipe for Few-Step Autoregressive Text-to-Video Generation

Abstract: Few-step autoregressive video generation enables low-latency streaming, yet prevailing training pipelines separate causal adaptation from distribution matching. Recent representation-based text-to-video methods retain this separation by starting from ODE-initialized checkpoints, even when the target distribution is defined directly by reference videos. We introduce Direct Forcing, a single-stage framework that uses reference videos both as paired examples of how a history continues and as a target population for self-generated rollouts. Causal Continuation Alignment learns paired spatiotemporal correspondence under reference histories, while Rollout Distribution Matching uses maximum mean discrepancy to align the representation distributions of self-generated and reference videos. Together, these complementary signals allow a pretrained video model to learn causal continuation and the target video distribution in a single training stage. Dynamic Gradient Balancing adapts the relative gradient contributions of continuation alignment and distribution matching during training. Starting from bidirectional Wan2.1-1.3B, Direct Forcing learns a four-step autoregressive generator in 250 updates without a separate initialization stage or online diffusion-score supervision, achieving 84.63 VBench Total and 86.01 Quality. Ablation studies further demonstrate the benefits of dynamic gradient balancing and reveal distinct representation preferences: deeper features benefit continuation alignment, whereas intermediate features yield better performance for rollout distribution matching.

Thu 22 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Generating videos step-by-step based on text descriptions can be slow if done in many steps. The authors propose a method called Direct Forcing that trains a video model in a single step to predict future frames quickly and accurately. It learns both how to continue a video from past frames and how to make the generated videos similar to real ones, using a balanced training approach. This method speeds up training and improves video quality without complicated setups.
Open → 2610.30861v1

Creative prompt expansion improves video and audio quality in generation

VideoGen Model 1: Prompt Expansion Versus Formatting in Audiovisual Generation

Abstract: Creative prompt expansion improved overall audiovisual quality scores over raw prompts in a controlled pilot of VideoGen Model 1, our post-trained version of MiniMax H3. We compared the raw, reformatted, and expanded versions of 30 prompts, holding the generator, seed, and sampling settings fixed. Each clip was rated separately on five dimensions. Their mean formed an overall score on a 1-10 scale. Compared with formatting alone, overall scores increased by 0.63 points (95% interval: [+0.11, +1.15]) under VA-Judger and 0.15 points (95% interval: [-0.03, +0.35]) under Gemini 3.8 Flash, whose overall change remained uncertain. Prompt-alignment scores increased by 0.97 points under VA-Judger and 0.50 under Gemini. VA-Judger also showed higher audio quality; visual-quality ratings changed little under either judge.

Wed 21 OctComputer Vision and Pattern Recognition
The gist
Generating videos from short text prompts can result in varying quality. The authors found that expanding the text prompts with creative details leads to better overall video and sound quality compared to using the original or just reformatted prompts. They tested this by keeping the generation method constant and measuring improvements using two different evaluators. The biggest gains were in how well the video matched the prompt and in audio quality, while visual quality showed little change.
Open → 2610.29407v1

Code2Video Compass benchmark evaluates code based video creation

Code2Video-Compass: A Comprehensive Benchmark for General-Purpose Code-Based Video Generation across T2V, I2V, and V2V

Abstract: Code-based video generation offers fine-grained control, targeted editing, and component reuse compared with direct video generation. This approach requires multimodal large language models (MLLMs) to translate visual and temporal requirements into executable programs that produce complete videos. However, existing code-based benchmarks primarily target specialized tasks, leaving complete video creation across diverse creative categories and input modalities insufficiently evaluated. We introduce Code2Video-Compass, a general-purpose benchmark for code-based video generation across text-to-video (T2V), image-to-video (I2V), and video-to-video (V2V) settings. Code2Video-Compass comprises 2,104 tasks spanning 10 categories, 24 subtypes, and three complexity dimensions covering construction, dynamic animation, and state dependency. It combines tasks curated from textual instructions and visual references with corresponding T2V, I2V, and V2V tasks derived from complete professional and user-created videos. We propose a dual-level evaluation framework combining 12 objective metrics and five subjective metrics. Objective evaluation covers operational performance, video quality, text-video alignment, and reference fidelity, while subjective evaluation uses an MLLM judge to evaluate instruction following and holistic video quality. Extensive evaluation of 15 recent proprietary and open-weight MLLMs from nine families reveals that even leading models face challenges in reliably generating high-quality complete videos aligned with textual and visual requirements, with performance generally declining as construction, dynamic animation, and state-dependency complexity increases. We anticipate that Code2Video-Compass will advance reliable, coherent, and controllable code-based video generation and inspire broader progress in programmatic visual creation.

Tue 20 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Creating videos by writing code instead of making them directly allows for more precise control and easier editing. The authors made a big test called Code2Video-Compass to see how well different AI models can create videos from text, images, or other videos using code. Their test covers many types of video tasks and checks video quality and how well the videos match the instructions. They found that even the best models still have trouble making good, fully accurate videos, especially when the tasks are more complex. This benchmark helps improve AI that makes videos through coding.
Open → 2610.28416v1

Multimodal model improves emotion recognition for oddly satisfying videos

PI-MLLM: Psychophysics-Inspired Multimodal Large Language Model for Oddly Satisfying Video Emotion Analysis

Abstract: Oddly Satisfying Videos (OSVs) can evoke strong viewer responses through repetitive motion, visual tactility, and ASMR-like sounds despite containing few explicit semantic emotion cues. This makes OSVs a challenging setting for Video Emotion Analysis (VEA), since most existing methods focus on semantic video content and under-model the physical sensory dynamics that shape viewer responses. To support Oddly Satisfying Video Emotion Analysis (OSVEA), we construct the Oddly Satisfying Video Dataset (OSVD), comprising 12,000 annotated 5-second clips. Each clip provides viewer-emotion labels under video-only, audio-only, and audio-visual conditions, together with explanations grounded in visual and acoustic sensory evidence. Building on OSVD, we propose PI-MLLM, a psychophysics-inspired multimodal large language model that integrates Modality-Specific Dynamics Alignment (MSDA) and Coalition-Based Audio-Visual Attribution (CAVA). MSDA uses auxiliary prediction objectives to align modality representations with visual motion and acoustic energy-onset dynamics. CAVA evaluates emotion scores across modality coalitions and applies Shapley attribution to identify the dominant modality and determine whether visual and auditory evidence is synergistic or conflicting. Experiments show that PI-MLLM improves both viewer-side emotion prediction and sensory-grounded explanation quality, suggesting the value of physical sensory cues to video emotion analysis.

Tue 20 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Oddly satisfying videos, like those with repetitive motions or soft sounds, affect viewers emotionally but don’t show clear feelings in their content. The authors built a special video collection with viewer emotions labeled by sound, visuals, or both to study this. They created a new AI model that looks at movement and sound patterns to better predict emotions and explain why. This model helps understand how physical sensations in videos influence viewer feelings.
Open → 2610.28264v1

Audio video consistency measured at event level improves generation quality

AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

Abstract: As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/

Thu 15 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
When computer programs create videos with matching sounds, it's hard to check if the sound and video really fit well together, especially for complex scenes with many events happening. The authors designed a new way to check consistency between sound and video by focusing on individual events in the scene rather than the whole clip. They trained a model called AVC-Reward that automatically judges how well audio matches video, and it matches human opinions better than previous methods. This tool can help improve the quality of audio-video generation by guiding the process to make better-matched scenes.
Open → 2610.22321v1

Unified multimodal model generates consistent images and text over long sequences

Scaling Unified Multimodal Models to Long-Horizon Interleaved Generation

Abstract: Unified multimodal models (UMMs) provide an interface for interleaved text-image generation, where generated visual states can directly inform subsequent reasoning and synthesis. However, existing systems are still primarily demonstrated on short trajectories or simplified visual content. We develop a large UMM natively trained on interleaved multimodal data, with strong capabilities in image synthesis, text rendering, complex composition, and multimodal planning, and study how to extend these native capabilities to substantially longer autoregressive trajectories. The main challenge is that generated images continuously accumulate in context: irrelevant or imperfect visual states interfere with future synthesis, while previously established characters and scenes must still be recovered after long intervals. We introduce a visual-memory framework that maintains persistent visual references, selectively retrieves scene-relevant historical states while preserving the complete semantic history, and uses few-step self-rollout to adapt the model to imperfect generated histories. We further introduce LongMIG-Bench for evaluating long-horizon text-to-multi-image and image-to-multi-image generation. Our model achieves high-fidelity long-horizon interleaved generation with strong character and scene consistency, demonstrating how visual memory can scale native interleaved generation to substantially longer trajectories.

Mon 12 OctComputer Vision and Pattern Recognition
The gist
It is hard for AI models that create both pictures and words to keep everything consistent when making long stories with many images and text steps. The authors built a big AI that can generate images and text back and forth for a long time without losing track of characters or scenes. They introduced a way to remember and retrieve important past images so future pictures stay consistent, even if some earlier ones were imperfect. They also created a test to measure how well models handle these long image and text sequences.
Open → 2610.17080v1

VoTSpeech separates voice design from speech generation in TTS systems

VoTSpeech: Decoupling Voice Design from Speech Generation in Instruction-Following TTS

Abstract: Instruction-following text-to-speech (TTS) typically leaves voice design implicit, as conventional speaker embeddings encode reference audio rather than text-driven targets. We propose VoTSpeech, which decouples voice design from speech generation via an explicit intermediate voice latent. Built on a shared language model backbone, VoTSpeech first predicts a continuous voice latent from text instructions using a flow-based module. This latent serves as a shared condition for both the language model and the Acoustic DiT, guiding autoregressive sequence modeling and acoustic latent generation, respectively, before an audio VAE decoder converts the acoustic latents into waveforms. To capture speaker identity and complementary acoustic nuances, the voice latent is jointly supervised by speaker embeddings and temporally pooled audio VAE features. With task-specific adaptation, VoTSpeech performs strongly on InstructTTSEval-ZH, demonstrating competitive capabilities against strong open-source models while maintaining high naturalness. These results confirm that explicit voice latent design and its conditioning interface effectively empower text-driven voice design for controllable speech synthesis. Code, model, and audio samples are available at https://github.com/ywbn/VoTSpeech .

Sat 10 OctSoundComputation and Language
The gist
Making computer voices that follow instructions is hard because voice features are mixed up with the way speech sounds are made. The authors created VoTSpeech, which first turns text instructions into a clear voice representation before turning that into speech sounds. This extra step helps the system better control how the voice sounds while speaking. Tests show VoTSpeech matches or beats other open systems and sounds natural.
Open → 2610.14694v1

Large vision language models stop seeing things that are not there

From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment

Abstract: Object hallucination remains a major obstacle for large vision-language models (LVLMs) to generate reliable content. An intuitive mitigation strategy is to suppress hallucination-related components in hidden representations. However, these components may also contain useful information, and suppressing them can weaken the model's multimodal capabilities. In this paper, we propose ResOT, a training-free method that repairs representations at inference time through localized distribution alignment. Specifically, ResOT projects dominant hallucinated directions away from the faithful subspace, forming a low-dimensional residual subspace for intervention. Within this subspace, ResOT uses Gaussian optimal transport (OT) to align the hallucinated distribution with the faithful one. The resulting map defines repair targets with minimal changes to the original representations. At inference, ResOT adaptively controls how far each token state moves toward its OT target. Experiments on three representative LVLMs show that ResOT substantially reduces object hallucination while improving image caption quality and multimodal performance across multiple benchmarks. Code will be released.

Thu 8 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Large vision-language models sometimes make mistakes by imagining objects that aren't really in the picture, called hallucinations. The authors introduce a way to fix this problem without retraining the model, by adjusting its internal representations at the time it makes predictions. They move the parts of the model's memory related to hallucinated objects closer to more accurate parts, reducing errors while keeping the model’s abilities strong. Tests show this method improves both accuracy and the quality of image descriptions.
Open → 2610.11826v1

Objective metrics better predict human judgments for co speech gestures

Perceptually Grounded and Semantics-Aware Evaluation for Holistic Co-Speech Gesture Generation

Abstract: Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations. Unlike Semantic Score (SC), which shows no significant association with the evaluated perceptual dimensions, SGP is selectively aligned with speech-aware human judgments. We construct five target-specific composite metrics aligned with the subjective dimensions. These composites improve perceptual alignment across all five dimensions, with the largest gains for absence of animation errors and content match, indicating that complementary objective signals can better approximate human judgments than individual metrics alone. Overall, our results show that objective metrics require validation against subjective evaluations.

Thu 8 OctComputer Vision and Pattern Recognition
The gist
Evaluating computer-generated gestures that accompany speech is tricky because existing measurements don't always match what people actually perceive. The authors created a new way to judge these gestures by combining various computer-based measurements with feedback from real users. They introduced a new metric that tracks how well the meaning of gestures is kept in generated movements. Their tests show that some new combined metrics better match what humans think, especially in timing and content accuracy. Overall, they emphasize the importance of checking automatic scores against human opinions.
Open → 2610.11669v1

Steerspeech lets text to speech voices express emotion with precision

Steerspeech: Activation Steering For Emotion Control In Generated Speech

Abstract: Pretrained text-to-speech (TTS) models can generate expressive speech, but reliable inference-time emotion control remains challenging: prompts and reference audio offer coarse, inconsistent control, whereas specialized conditioning and model adaptation require costly training. We present SteerSpeech, a lightweight activation-steering framework that controls emotion by injecting steering vectors into hidden activations. For each target emotion we train a lightweight low-rank transform, using a multi-expert objective that encourages monotonic emotion control while preserving speaker identity and linguistic content, constraining steering drift, and keeping the TTS backbone frozen. To optimize through discrete speech tokens, we introduce a two-pass generation-and-replay pipeline using a straight-through estimator to backpropagate expert supervision through sampled tokens. At inference, a target-emotion steering direction is optimized with its respective transform and injected into the base TTS model. Objective and subjective evaluations with Qwen3-TTS across seen, unseen, and accented speakers show stronger continuous emotion control with limited speaker and content degradation. SteerSpeech achieves 1.08x-7.12x baseline target-emotion scores and for a representative emotion subjectively, it receives 78.1%-96.8% intensity preference and 1.43x-1.46x speaker-identity preservation at high steering strengths.

Wed 7 OctSoundMachine Learning
The gist
Text-to-speech systems can speak with emotion, but controlling the feeling consistently and clearly is hard. The authors present SteerSpeech, a way to finely adjust emotions like happiness or sadness in speech by tweaking the model’s internal signals without retraining everything. This method works quickly and keeps the voice and words clear. It improves emotional expression across different voices and languages compared to earlier methods.
Open → 2610.10415v1

Salt++ improves few-step streaming video and audio generation quality

Salt++: Context-Aligned Post-Training for Few-Step Streaming Multimodal Generation

Abstract: Few-step streaming audio--video generation requires both causal modeling and step distillation, yet standard training recipes face two context-related challenges. Teacher forcing pairs clean history with a noisy target, but supervises predictive contextual representations only indirectly through velocity prediction. Meanwhile, directly reusing bidirectional score models in causal Distribution Matching Distillation (DMD) creates a mismatch between generation and scoring contexts. We address these challenges with Salt++, a two-stage post-training framework comprising Causal Self-Flow (CSF) and context-aligned autoregressive DMD. CSF exploits contextual information asymmetry by varying the history while keeping the noisy target fixed: a noise-mixed-history student aligns its intermediate representations with those of a clean-history exponential-moving-average teacher. This self-supervised signal encourages the student to extract semantic information and improves cross-modal alignment. Context-aligned AR DMD shares the causal mask and prefix across generator sampling, fake-score training, and real-score evaluation to match generated and reference distributions under a block-conditional KL objective. With calibrated teacher guidance, it performs clean-prefix few-step distillation and then adapts to generated histories without switching objectives or requiring separate consistency distillation. At 480p, Salt++ improves visual and motion quality by 57% and 45% over OmniForcing on JavisBench under the same 4-step causal setting. A separate scale-wise post-training stage extends Salt++ to 4-step $1664\times960$ generation, outperforming bidirectional LTX-2 on six of seven reported metrics. Project page: https://xingtongge.github.io/Saltpp

Tue 29 SeptComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Generating audio and video in a few quick steps is hard because the system needs to understand both past and future information differently. The authors present Salt++, a new approach that trains models to better handle these challenges by aligning how the system learns from past context and noisy new data. This makes the generation process more accurate and better aligned across sound and visuals, improving quality significantly. They tested Salt++ on video benchmarks and showed it works better than previous methods.
Open → 2609.36995v1

Joint audio video generation improves fidelity and sync with new RL method

AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation

Abstract: Recent years have witnessed major progress in joint audio-video generation. Existing models still suffer from limited per-modality fidelity, insufficient text-modality alignment and weak cross-modal synchronization. While reinforcement-learning post-training offers a promising remedy, directly adapting it to joint audio-video generation is challenging. Heterogeneous multimodal rewards entangle learning signals and complicate credit assignment. Joint optimization of two modality towers is computationally expensive given their divergent dynamics. Moreover, synchronization evaluation difficulty depends on paired samples, preventing fair reward comparisons. We propose AV-GRPO, a modality-anchored online diffusion RL framework, and 5DAV, a decoupled, difficulty-controllable training dataset. AV-GRPO includes three key modules: (1) modality-anchored rollouts to disentangle learning signals and stabilize difficulty; (2) trajectory-locked frozen-tower optimization to reduce cost and reassign credit; (3) adaptive objectives and perturbation strengths tailored to modality-specific dynamics. This converts coupled multimodal preference learning into unimodal subproblems for precise reward attribution and better synchronization. Our 5DAV dataset decouples samples across five dimensions for systematic training. Experiments on JavisBench and VABench demonstrate AV-GRPO outperforms LTX-2.3 in generation quality, semantic alignment and cross-modal synchronization under LoRA and full fine-tuning. Ablations confirm our designs. Code and data: https://github.com/zhiyuxu03/AV-GRPO

Thu 24 SeptComputer Vision and Pattern RecognitionSound
The gist
Generating videos with matching sounds is hard because models struggle to make each part realistic and well-timed together. The researchers created AV-GRPO, a new learning approach that separates audio and video training to give clearer feedback and better timing. They also made 5DAV, a special dataset to help train models more effectively. Tests showed their method produces better quality videos and sounds that match well with the text descriptions and each other.
Open → 2609.29816v1