Papers for

multimedia content platforms

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Video understanding improves with smart adaptive frame analysis

EVAThink: Evidence Sufficiency-Guided Training for Adaptive Thinking in Video Understanding

Abstract: Under a fixed frame budget, video understanding with vision-language models (VLMs) depends on whether the sampled frames provide sufficient evidence for the question. Sufficient frame evidence supports direct answering, whereas less informative observations may benefit from reasoning that connects the available cues. We introduce EVAThink, which enhances video understanding through evidence sufficiency-guided training for adaptive thinking. First, Recoverable Evidence Gap Mining constructs evidence-dense and evidence-sparse video views that share the same question and answer under an identical frame budget. It retains pairs where sparse views yield lower direct-answer accuracy than dense views but benefit from explicit reasoning, providing supervision for response-mode selection. Second, Evidence-Guided Two-Stage Reinforcement Learning uses stage-specific rewards to guide training with evidence-based mode preferences and then refine response efficiency, while first-token advantage regularization mitigates mode collapse. Trained on only 6.7K curated samples, EVAThink consistently improves upon the base model across video understanding benchmarks under both 32- and 64-frame budgets, while outperforming strong baselines on average. The code, model, and datasets are available at \href{https://github.com/ChangtiWu/EVAThink}{EVAThink}.

Tue 10 NovComputer Vision and Pattern Recognition
The gist
Understanding video content often depends on the key frames selected for analysis, but sometimes the frames don't provide enough clues for direct answers. The authors developed a method called EVAThink that teaches a video-language model when to rely on direct answers and when to use deeper reasoning based on how much useful information is present. This approach uses specially chosen video clips to train the model to adapt its thinking style, improving its performance on video question-answering tasks with limited frames. Results showed better accuracy even with fewer frames, suggesting smarter frame use helps computers understand videos more like people do.
Open → 2611.13907v1

Contrastive audio language models improve with larger diverse datasets

LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining

Abstract: Recent progress on contrastive audio-language models can be attributed mainly to the scale and quality of their training data. Larger training datasets expose models to a broader range of sounds and corresponding audio captions. To exploit this, we collect 1.48 billion audio samples from academic datasets, internet datasets, web collections, Common Crawl, and YouTube, and process them with a multi-stage curation pipeline. Furthermore, we generate short and long captions for samples in our corpus that lack them. We use our dataset to study the scaling behavior of contrastive audio-language models based on the NaFlex architecture for the audio encoder. Our experiments identify trade-offs between model size and training data for a given compute budget. Moreover, our resulting models, CLAP2, outperform prior audio-text models of comparable size on the MAEB and HEAR benchmarks. We release our dataset, captions, model checkpoints, and code.

Sun 8 NovSoundComputation and Language
The gist
Understanding sounds and matching them with text descriptions is a tricky problem. The authors gathered over a billion diverse audio clips from many sources and cleaned them carefully. They added captions when missing and trained models that connect sounds and words better than before. By studying how bigger models and more data affect results, they found good trade-offs for performance. Their trained models beat previous ones on popular benchmarks and are shared for others to use.
Open → 2611.11592v1

Visual token pruning improves efficiency of multi image vision language models

RI-Prune: Relation and Information Aware Visual Token Pruning for Multi-Image Vision-Language Models

Abstract: Large vision-language models (LVLMs) have recently been extended to handle multiple input images, but this extension dramatically increases the visual token budget, making visual-token pruning not merely an efficiency option but a practical necessity. Existing visual-token pruning methods, primarily designed for the single-image setting, rely on attention-derived saliency signals to estimate per-token importance. However, this attention-centric design transfers poorly to the multi-image regime: it causes substantial performance degradation on multi-image benchmarks, suffers from strong positional bias, and loses discriminative attention signals as the number of input images increases. To test whether attention-selected tokens are truly important for multi-image reasoning, we conduct two complementary causal probes Leave-One-In and Leave-One-Deletion, which measure token importance by restoring individual tokens to a zeroed visual sequence or removing them from the full sequence. Attention-based pruners provide only marginal gains over random selection, showing that attention is a weak proxy for visual-token importance in multi-image reasoning. Motivated by this gap, we propose Relation and Information Aware Pruning (RI-Prune), a training-free, attention-free pre-decoder selector for multi-image visual token pruning. RI-Prune scores each visual token using a z-normalized combination of three signals computed from projector outputs: query relevance, cross-image correspondence with compact per-image prototypes, and intra image representativeness. By integrating these signals, RI-Prune performs balanced per-image Top-K selection to preserve information across input images. Across multiple multi-image benchmarks and three LVLM backbones, RI-Prune achieves a stronger trade-off between token reduction and task performance than prior pruning methods, particularly under aggressive token budgets.

Wed 4 NovComputer Vision and Pattern Recognition
The gist
Handling multiple images at once in large vision-language models creates too many visual tokens, which slows down processing. Existing methods to reduce tokens rely on attention signals that work well for single images but fail for multiple images, causing performance drops. The authors tested and found attention is a weak way to pick important tokens for multi-image tasks. They propose RI-Prune, a new method that scores tokens using multiple signals related to their importance and relations across images, without needing training or attention. This method keeps more useful information while reducing tokens, improving speed and accuracy on multi-image benchmarks.
Open → 2611.06047v1

Video language models reduce hallucinations using attention based perturbations

Mitigating Video Hallucinations via Attention-Guided Representation Perturbation

Abstract: Video large language models (Video LLMs) often generate responses unsupported by the input video. Contrastive decoding (CD) can mitigate these hallucinations, but its effectiveness depends on how the negative branch weakens visual evidence. Existing methods rely on predefined perturbations or masking guided by question reconstruction, offering limited control over the model's visual reliance during answer generation. We introduce Attention-guided Representation Perturbation for Contrastive Decoding (ARP-CD), a training-free framework for constructing contrastive negatives directly in representation space. ARP-CD uses the attention mass assigned to visual tokens at the answer-prediction position to guide bounded gradient updates of the language model's input visual hidden states. These updates reduce visual attention and produce a negative input, whose predictions are contrasted with those of the original branch throughout generation. Experiments across four Video LLMs and four benchmarks show that ARP-CD outperforms representative training-free baselines in most model--benchmark settings, reducing hallucinations while improving general video understanding. Further analyses show that attention-guided perturbation directions improve negative quality and that the resulting negatives provide effective contrastive references for correcting the original predictions.

Wed 14 OctComputer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
The gist
Video large language models sometimes give answers not supported by the video they analyze, which is called hallucination. The authors introduce a way to guide the model's focus away from visual parts it relies on too strongly by slightly changing its internal video representations. This creates a negative example that helps the model contrast good answers with less truthful ones during generation. Their method, tested on several models and benchmarks, reduces hallucinations and improves overall video understanding without needing extra training.
Open → 2610.19728v1

Hybrid method improves video action detection and event understanding

HyrCap: Hybrid Rank-Calibration of Action Proposals for Temporal Event Understanding

Abstract: Temporal action detection in long untrimmed videos suffers from inaccurate localization and low-quality proposals, especially with sparse annotations and ambiguous boundaries. For open-ended video understanding and LLM-driven event analysis, reliable proposals connect temporal localization, event annotation, and semantic understanding. However, conventional temporal action detection (TAD) pipelines lack explicit modeling of proposal localization quality, cross-representation support, and inter-proposal relations. We propose Hybrid Rank-Calibration of Action Proposals (HyrCap), a proposal-centric query-based TAD framework for temporal event understanding. Hybrid combines complementary video features with different localization cues, while Rank-Calibration adjusts proposal ranking confidence to reflect localization reliability and candidate competition. HyrCap treats dense query-based predictions as an over-complete temporal event hypothesis space. Cross-Representation Consensus Calibration, Localization Quality Estimation, and Proposal Redundancy Suppression improve proposal ranking reliability, localization quality, and candidate discrimination. Inspired by LLMs' capabilities in fine-grained semantic recognition, event description, and reasoning, we introduce HyrCap for Semantic Action Understanding, using calibrated proposals for standard TAD and as inputs to LLM-based semantic understanding. We define a staged evaluation protocol to separately measure event localization, segment-level semantic understanding, and combined event understanding performance. Experiments on public benchmarks show that HyrCap achieves state-of-the-art performance across multiple metrics and provides a finer-grained paradigm and evaluation protocol for query-based event segmentation and annotation.

Wed 14 OctComputer Vision and Pattern Recognition
The gist
Detecting when actions happen in long videos is hard because the boundaries between events can be unclear and annotations are often sparse. The authors present HyrCap, a new approach that better ranks and calibrates action proposals by combining different video features and carefully estimating how accurate each proposal is. This method improves the quality and reliability of detected events and supports deeper understanding by using language models for event description. Their experiments show better results than previous methods and introduce a new way to evaluate both localization and semantic understanding of actions.
Open → 2610.19526v1

Improving audio understanding with reinforcement learning on long recordings

ATTEST: Evidence-Grounded Reinforcement Learning for Long-Audio Temporal Understanding

Abstract: Temporal grounding is fundamental to audio understanding, yet large audio language models (LALMs) remain weak at it on long audio, where hallucinations compound and annotated data is scarce. Existing benchmarks cover only short clips or offer coarse, synthetic annotations. We introduce EgoSCAPE, a manually verified benchmark created from Ego4D with 700 real audios of durations up to 5 minutes, annotated with dense captions, sound sources, audio timelines, and temporal grounding queries. We further propose ATTEST, a GRPO-based reinforcement learning framework that improves temporal grounding without changing the base model's architecture or token vocabulary. ATTEST combines (i) a counterfactual grounding reward penalizing predictions that survive removal of their acoustic evidence, (ii) an attention-density reward encouraging generation to attend to its own claimed time spans, and (iii) token-wise advantages giving segment-level credit for long dense captions. ATTEST outperforms prior temporal grounding methods on existing dense captioning and grounding benchmarks, with fewer hallucinations. Together, EgoSCAPE and ATTEST establish a rigorous evaluation-training recipe that brings temporally grounded audio understanding closer to practical, long-form deployment.

Mon 12 OctSound
The gist
Understanding when events happen in long audio recordings is hard because current models often imagine details that aren't there and data with accurate timing is rare. The authors created a new benchmark with nearly 700 real-world videos up to 5 minutes long, carefully labeled with sound descriptions and timing information. They also developed a new training method called ATTEST that teaches models to be more accurate about timing without changing their basic setup. This method reduces false details and improves performance on long audio tasks, making it easier to handle real-life long recordings.
Open → 2610.17149v1

Spatio-temporal video grounding models often ignore query relevance

Investigating Query-Insensitive Behavior in Spatio-Temporal Video Grounding

Abstract: Spatio-temporal video grounding (STVG) aims to localize objects or events described by natural language queries in both space and time. Existing STVG models are typically trained and evaluated under the assumption that each query is relevant to the input video. In this work, we challenge this assumption by studying the behavior of state-of-the-art STVG models under irrelevant queries and missing textual input. Our experiments show that current models can still produce plausible spatio-temporal predictions even when the query is unrelated to the video or removed entirely. We further analyze HCSTVG-v2 and VidSTG to identify dataset regularities that may encourage such query-insensitive behavior. Our study highlights an underexplored limitation of STVG models and motivates negative-aware evaluation protocols and architectures that explicitly assess query relevance.

Mon 5 OctComputer Vision and Pattern RecognitionArtificial IntelligenceComputation and Language
The gist
Spatio-temporal video grounding means finding objects or events described by words in a video, both in where and when they happen. This paper finds that current models still guess locations and times even if the words do not match the video or are missing. The researchers investigated why this happens and found issues in the data that cause models to ignore the query. They suggest new ways to test and build these models so they pay attention to whether the words really relate to the video.
Open → 2610.06018v1

EviDETR improves video search and highlight spotting from text queries

EviDETR: Preserving Query-Relevant Temporal Evidence for Moment Retrieval and Highlight Detection

Abstract: Joint video moment retrieval and highlight detection requires identifying query-relevant temporal segments while estimating clip-level saliency, yet DETR-style pipelines do not explicitly preserve query-relevant evidence throughout encoding, decoding, and cross-task prediction. We propose EviDETR, an evidence-preserving framework with three components. Semantic-aware Feature Reweighting (SFR) enhances query-relevant clip representations through saliency estimation and cross-modal interaction. A Temporal Top-2 Mixture-of-Experts (TTop2MoE) decoder performs query-adaptive refinement via sparse expert routing. MR-to-HD (MR2HD) fusion transfers span-level retrieval evidence to clip-level highlight prediction through confidence-weighted multi-scale aggregation. Using CLIP+SlowFast features, EviDETR achieves 69.29 R1@0.5, 54.77 R1@0.7, and 48.41 Avg. mAP for moment retrieval on QVHighlights, together with 41.83 HD-mAP and 68.33 HIT@1. Strong results on TACoS and Charades-STA further demonstrate cross-dataset transferability.

Fri 25 SeptComputer Vision and Pattern Recognition
The gist
Finding specific moments in videos and spotting highlights based on a text query is tricky because existing methods don’t always keep track of what parts of the video relate to the query. The authors propose EviDETR, a new system that better remembers relevant video segments during analysis and prediction. It uses special techniques to focus on important clips and combines information effectively to improve both search and highlight detection. EviDETR shows strong performance on several video datasets, making it easier to find and highlight moments in long videos.
Open → 2609.30724v1