Papers for

security monitoring operators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

EARS uses sound and motion to pick key video frames for analysis

EARS: Evidence-Aware Retrieval via Sound and Motion for Budgeted Video Reasoning

Abstract: Uniform frame sampling can miss brief evidence in sparse-event videos when the VLM sees only K frames. We propose EARS (Evidence-Aware Retrieval via Sound and motion), a training-free, query-free front-end that ranks frames using optical-flow motion and audio energy, then applies temporal non-maximum suppression for diversity. The evidence score alone has frame-level AP 0.39; used only to select frames for the VLM, EARS with temporal diversity reaches localization AP 0.94, compared with 0.88 under Uniform sampling. In the updated matched study, at K = 1 EARS reaches XD-subset F1 0.660 versus 0.614 for centered Uniform (the paired interval includes zero); earlier backbone studies use separate protocols. On mostly-silent UCF-Crime, however, EARS slightly trails Uniform. A single configuration is used across both benchmarks; the new controls do not repeat the original held-out search. EARS therefore targets low-budget videos whose salient evidence aligns with motion or audio; on dense narratives or stealthy events, uniform or semantic selection remains preferable, and cheap modality statistics cannot reliably route between them. Code: https://github.com/mikeslin-ir/EARS

Mon 26 OctComputer Vision and Pattern Recognition
The gist
Watching every part of a video can miss brief but important moments if you only look at a few frames. The authors propose EARS, a simple way to pick the most interesting frames by looking at movement and sound instead of just regularly spaced frames. This method improves how well a video language model can find important moments in videos with noticeable action or noise, but it is less effective for quiet or dense videos. EARS works without extra training or searching, making it a lightweight way to select frames for video understanding.
Open → 2610.35536v1

Industrial system scales robot fleet memory by encoding only on demand

Industrial ReMEmbR: Scaling Spatiotemporal Memory to Robot Fleets

Abstract: Warehouse robot fleets continuously observe operational conditions, but preserving these observations as reusable semantic memory incurs substantial computation, communication, and storage costs. For example, explaining recurring aisle blockages can require reviewing video from multiple robots across several shifts. We present IndustrialRemembr, a shared spatiotemporal memory system that extends NVIDIA ReMEmbR to robot fleets and reduces server-side computation by generating scene descriptions only on demand. On each robot, an NVIDIA Jetson encodes video clips into retrieval embeddings with Cosmos-Embed1, vision tokens with Cosmos3-Edge, and reconstruction latents with the Wan2.2 VAE, without uploading raw camera frames. The encoding models are re-used to detect people in image patches, zeroing them out to support PII abstraction for GDPR-compliant deployments. We split Cosmos3-Edge at its vision-language boundary, placing the vision encoder on the robot and the language decoder on an on-premises server. A Nemotron agent retrieves memories by content, position, and time and queries the decoder using cached vision tokens. This design eliminates caption generation at write time, while cached-token decoding reproduces the unsplit model's output in all 450 tested decodes.

Tue 20 OctRoboticsArtificial IntelligenceSoftware Engineering
The gist
Warehouse robot fleets record lots of video to understand their environment, but saving all this data is costly. The authors designed IndustrialReMEmbR, a system that lets robots keep useful memory of their surroundings without sending raw video to central servers. Each robot creates smaller summaries and hides personal information to comply with privacy laws. The system delays creating detailed scene descriptions until needed, reducing computing work on the servers. This approach helps explain events like blocked aisles by searching through useful robot memories efficiently.
Open → 2610.28624v1

Optimal transport improves frame selection for long video understanding

A Principled Optimal Transport Framework for Frame Selection in Long Video Understanding

Abstract: Long-video understanding with multimodal LLMs is fundamentally constrained by the mismatch between video length and the model's limited visual context budget, making frame selection essential for practical reasoning. Existing query-aware methods typically rely on heuristic pipelines: they assign one-dimensional query-conditioned scores to frames and then apply sampling, ranking, or extraction rules to form the final subset. While effective in practice, such methods are generally not derived from an explicit optimization objective and struggle to capture the heterogeneous evidence required for complex reasoning. In this paper, we revisit long-video frame selection from a principled optimization perspective. We formulate it as a structured evidence-allocation problem and propose OTFS, an optimal transport framework that maps multiple evidence sources onto video frames. To reflect the asymmetry of this problem---important evidence should be preserved, whereas redundant frames need not absorb mass---we instantiate this view with a semi-unbalanced entropic optimal transport objective and an efficient solver, OTFS-Sinkhorn. Combined with an exact dynamic program for coverage-aware subset extraction, OTFS yields a practical two-stage training-free pipeline. We further show that, in the single-source case, our formulation induces a Gibbs-type distribution over frames, making Q-Frame's temperature-scaled distribution a special case of our framework while also providing a broader perspective for understanding related prior methods. Experiments on three long-video understanding benchmarks show that OTFS consistently outperforms strong training-free baselines. Codes are available at https://github.com/barry664/OTFS.

Tue 20 OctComputer Vision and Pattern Recognition
The gist
Long videos can be hard for AI models to understand because the models can only look at a limited number of frames. The authors looked at how to pick the most important frames for understanding these videos. They created a new method that uses a math idea called optimal transport to fairly assign importance from different clues to the frames. This method works better than older ways because it carefully balances keeping key information and ignoring redundancy. Their tests on video tasks show their approach picks frames that help the AI understand videos more accurately.
Open → 2610.27692v1

Atlas improves video object segmentation by adapting language model attention

ATLAS: Learning to Adapt LVLM Attention for Reasoning Video Segmentation

Abstract: Reasoning Video Object Segmentation (RVOS) aims to localize and segment objects referred to by natural-language expressions throughout a video. Although large vision-language models (LVLMs) implicitly encode localization cues in their attention maps, these attentions are often noisy, diffuse, and scattered across layers, heads, and modalities, limiting their direct use for segmentation. Existing methods typically rely on handcrafted attention-selection heuristics or computationally expensive attention refinement. We propose ATLAS, an attention adaptation framework that learns to transform raw LVLM attentions into object-centric localization cues. ATLAS treats attentions from all layers, heads, and modalities as structured features, combining high-resolution frame-level attentions that capture spatial information with low-resolution video-level attentions that encode temporal context. A lightweight U-Net equipped with temporal attention and FiLM-based modulation fuses these complementary representations to generate coarse segmentation heatmaps, which are propagated across the video using SAM2. Extensive experiments on challenging referring and reasoning video segmentation benchmarks show that ATLAS outperforms existing frozen-backbone methods. Code and models are available at \url{https://hamidreza-136.github.io/atlas/}.

Sat 17 OctComputer Vision and Pattern Recognition
The gist
Videos often have objects that people want to find and highlight based on spoken or written descriptions, but existing tools get confused because the clues from language models are scattered and unclear. The authors developed ATLAS, a system that learns to clean up and combine these clues from lots of parts of the language and vision models to make better guesses about where objects are in videos. It uses a special network to merge details over time and space and then sharpens the results with another tool to get clear object outlines throughout a video. Their tests show ATLAS works better than other methods that don’t change the base model.
Open → 2610.24783v1

CoWorld Reasoner improves video based visual reasoning across tasks

CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness

Abstract: As Vision-Language Models (VLMs) advance, they are increasingly expected to perform spatiotemporal reasoning over motion, actions, and view changes beyond what the input images show. Video World Models (VWMs) provide a natural unified engine for visual imagination by generating task-relevant visual content as additional evidence. However, existing approaches typically invoke a VWM through fixed pipelines centered on a predefined form of visual imagination, such as novel-view synthesis. To support flexible visual imagination for diverse reasoning tasks, we introduce CoWorld Reasoner, an agentic framework that uses a VWM as a unified visual reasoning engine. Its Video World Model-Driven Harness exposes multiple visual operations through tools and uses skills to guide the VLM in deciding whether to imagine, selecting an operation, constructing a generation request, and interpreting the resulting visual evidence. We further introduce VLM-VWM Harness Co-Evolution, which jointly revises tools, skills, and execution code based on execution feedback while keeping both model weights fixed. Across seven benchmarks spanning spatial intelligence, temporal and causal reasoning, and perceptual reasoning, CoWorld Reasoner outperforms direct reasoning with the same VLM. The strongest variant improves on all seven benchmarks, including relative accuracy gains of 11.8% on MMSI and 13.0% on ERQA, and outperforms all five evolving baselines using the same backbone across five benchmarks.

Thu 15 OctArtificial Intelligence
The gist
Understanding what's happening in videos involves tracking motion, actions, and changes in view. The paper presents CoWorld Reasoner, a system that uses a video world model to imagine and create new visual clues to help answer questions about videos. This system can choose different ways to generate helpful images and use them to reason better than just looking at the original video. The authors show that this approach improves performance on several tests involving spatial, temporal, and causal reasoning.
Open → 2610.21910v1

Active evidence acquisition improves decision making under cost constraints

ActiveEvidence: Decision-Focused Evidence Graphs with Verifiable Acquisition Gaps

Abstract: A multimodal agent must decide whether visual evidence is worth acquiring, which candidate or resolution to query, and when further interaction can no longer change its decision. We study this problem under a strict free-metadata/paid-observation boundary, stochastic vector costs, and a frozen execution trace, and treat it as decision-certifiable active evidence acquisition. A Counterfactual Evidence Interaction Graph (CEIG) represents unary evidence value, signed second-order interaction, and directed conditional harm while maintaining explicit score enclosures for unresolved interactions; a decision-weighted enclosure objective tightens the bounds that can change action identity. Gap-directed refinement evaluates only interactions that can displace the incumbent acquisition and stops once an $ε$-decision certificate shows that unresolved terms cannot change the selected action beyond a declared tolerance. The certificate is model-relative: it verifies policy-consistent optimality under the pinned learned score, and semantic quality is measured separately through post-freeze labels. Paired continuation rollouts learn the downstream value of paid previews, an online operational cost filter is separated from independent finite-policy-family trajectory certification, and an acquisition monitor with an independently implemented verifier binds every certificate to the frozen trace. On 640 cases split into 384/128/128 training, validation, and test cases, the controller reaches task score 0.842, grounding F1 0.816, evidence recall 0.873, 1.74 accepted calls, 286.5K visual tokens, and 742ms p95 latency. Dense replay of issued certificates shows 0.992 action agreement and a 0.988 certificate rate while evaluating 19.6 instead of 191.4 expensive interaction terms per decision.

Wed 14 OctArtificial IntelligenceComputer Vision and Pattern RecognitionMachine Learning
The gist
Deciding when and what visual evidence to gather can be costly and complex for AI agents. The authors present a framework that smartly chooses which evidence to acquire and when to stop gathering more, ensuring decisions stay reliable without unnecessary expenses. They achieve this by using a special graph to model evidence interactions and certify that no better choice exists beyond a certain tolerance. Their method was tested on many cases, showing it reduces work while keeping decision quality high.
Open → 2610.20532v1

DeltaStreamer improves video understanding by remembering changes and replaying details

DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding

Abstract: We propose DeltaStreamer, a novel agentic framework for streaming video understanding inspired by how humans observe, remember, and investigate. Streaming video understanding requires a system to retain evidence before knowing which details future queries will need. Existing approaches often obscure event transitions and omit relevant details when compressing video history, and cannot recover missing visual evidence. DeltaStreamer addresses these limitations by combining change-aware memory with query-driven planning and targeted evidence gathering with replay. Before a query arrives, it continuously observes the stream and explicitly records visual changes in structured memory, preserving how events unfold rather than only what is present. When a query arrives, a query-driven planner identifies the required evidence and guides memory retrieval and targeted replay. By revisiting historical frames, the agent can recover visual details omitted during observation and ground its answer in evidence beyond the retained descriptions. This connects continuous observation with active investigation tailored to each question. Experiments on OVO-Bench and StreamingBench demonstrate improvements over representative streaming video understanding methods, with overall scores of 66.07% and 59.53%, respectively. Project page: https://bbbbchan.github.io/DeltaStreamer/

Wed 14 OctComputer Vision and Pattern RecognitionArtificial IntelligenceHuman-Computer Interaction
The gist
Understanding videos as they play is tricky because you don’t know what questions will come later. The authors built DeltaStreamer to watch videos like humans do: by noticing what changes, remembering events, and then revisiting parts of the video when asked a question. This helps the system not miss important details and answer questions better. Tests show DeltaStreamer does better than other methods on video understanding challenges.
Open → 2610.19849v1

Complex fusion improves recognizing unseen audio visual events in videos

Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

Abstract: Open-Vocabulary Audio-Visual Event Localization (OV-AVEL) labels each video segment with an event class, including classes that were never seen during training. The dominant pipeline uses a frozen multimodal foundation model (e.g. ImageBind) to embed the visual frame, the audio mel-spectrogram, and each candidate class name into a shared space, then computes two cosine similarities for each segment against each class: visual-text and audio-text. Existing methods then collapse this pair into a single scalar score with a fixed rule (geometric mean, weighted average) before taking the argmax. Instead, we compute complex-valued similarities and learn their fusion using a complex-valued neural network (CVNN). Each modality's standard representation becomes the real part of our pipeline, and a paired companion stream supplies the imaginary part. We use imaginary part of iHSV for visual modality and CycleGAN-translated phase spectrogram for audio modality as these companion streams. This results in two complex similarities, which are then fused. While the vision and audio encoders remain frozen, only the temporal-attention blocks and the fusion CVNN are trained. The four-stream complex architecture sets a new state of the art on both OV-AVEL benchmarks. On the open (unseen-class) split of OV-AVEBench we reach 66.5/59.1/54.1% Acc/Seg-F1/Event-F1 (+1.6/+4.1/+6.6 over the previously reported fine-tuned baseline), with consistent gains for seen classes as well. We also modify AVE dataset for this task and observe that our architecture reaches 60.7/51.9/50.4% Acc/Seg-F1/Event-F1, achieving state-of-the-art OV-AVEL results on it as well. We also propose a two-stream alternative, which also sees great improvements over the baseline.

Thu 8 OctComputer Vision and Pattern RecognitionMachine LearningMultimedia
The gist
This paper addresses the challenge of identifying events in video segments, even those events not seen before by the system during training. The authors propose a method that combines sound and image information in a special way using complex numbers, rather than just combining scores with simple math. They keep the main image and audio models fixed and only train parts that focus on combining these signals. Their approach improves accuracy on standard tests that measure how well systems can detect events based on both visuals and audio.
Open → 2610.11846v1

VidHarness automates video agents for cheaper long video analysis

VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding

Abstract: Vision-language models (VLMs) can answer questions about hour-long videos, but processing every frame is prohibitively expensive, even though the evidence for a question usually spans only a few seconds. Video agents, i.e., harness programs wrapped around a frozen VLM, address this by observing the video selectively, yet existing harnesses are hand-crafted by experts through slow build-and-test cycles. We propose VidHarness, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment. To escape the local optima of greedy refinement, we organize the evolution as Monte Carlo tree search (MCTS), and to reduce the evaluation cost, we integrate uncertainty-aware multi-fidelity validation, which screens new harnesses on a few questions and promotes only the promising ones. Since the best harness varies with the frame budget, we further introduce a mixture-of-harness that routes each question to a harness specialized for its budget. VidHarness sets new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforms the strongest hand-crafted video agent by up to $11.2$ points, and generalizes to the knowledge-intensive benchmarks Video-MMMU and MMVU with fewer than half of the frames of uniform sampling.

Tue 29 SeptComputer Vision and Pattern Recognition
The gist
Looking at every moment in hour-long videos to answer questions is too slow and costly. The authors created VidHarness, a system that automatically designs smart programs called harnesses to watch only important video parts. It uses a search method to find the best way to watch videos efficiently, saving processing time and improving accuracy. This approach works better than expert-made methods and adapts to different budgets for analyzing video frames.
Open → 2609.38413v1

Video agents improve understanding by revising their observation methods

Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution

Abstract: Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use those observations. However, execution traces contain only the evidence acquired by the current harness, leaving competing explanations for failure unresolved and limiting the basis for self-improvement. We introduce Video-RSI, a framework for recursive self-improvement in which a video understanding agent uses its own language model to revise its harness. Through active video investigation, the model revisits the original training videos to test competing failure explanations with additional observations, grounding proposed changes in evidence beyond the existing trace. Cost-aware harness evolution turns these diagnoses into reusable revisions and determines which revisions to retain by considering both answer accuracy and visual cost. Across our evaluation settings on video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames and achieves competitive accuracy-efficiency trade-offs against existing video understanding agents. These results demonstrate the potential for video understanding agents to improve their own evidence acquisition and use through harness evolution. Code is available at https://github.com/bingjunluo/Video-RSI .

Tue 29 SeptArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Video understanding systems learn by watching videos and following a set method for what to look at and analyze. The authors created a way for these systems to improve themselves by changing their own methods using their language skills, allowing them to gather better evidence when videos don’t make sense. This self-improvement also balances the need for accuracy and the amount of video watched, so the system can be more efficient. Tests showed the improved systems were more accurate and used less video than before, competing well with other methods.
Open → 2609.37950v1

MetaSampling reduces frames for better long video question answering

MetaSampling: Making Frame Samplers Efficient for Long-Video Question Answering

Abstract: Frame selection is an important component of long-video question answering (VQA) with Multimodal Large Language Models (MLLMs). Existing frame-selection methods improve over simple top-$k$ embedding retrieval and uniform sampling, but are typically applied under a fixed global selection budget. We introduce \textbf{MetaSampling}, a training-free, plug-and-play sampling strategy that can be applied on top of existing frame selectors. MetaSampling improves downstream VQA efficiency by dynamically reducing the number of frames passed to the MLLM while preserving, and in some cases improving, answer accuracy. We evaluate MetaSampling across 36 paired frame-selector--MLLM-backbone--VQA-benchmark configurations. MetaSampling reduces the number of selected frames in all 36 configurations and improves accuracy in 25 of them, yielding an average frame reduction of $8.9\%$ while slightly improving accuracy overall.

Sun 27 SeptComputer Vision and Pattern Recognition
The gist
Answering questions about very long videos can be hard because a computer has to look at many video frames to understand what's happening. The authors created MetaSampling, a method that helps choose fewer important frames to show the computer without losing accuracy in the answers. MetaSampling works on top of other frame selection methods and can sometimes even improve how well the computer answers questions. It was tested with many different video question answering systems and consistently reduced the number of frames needed.
Open → 2609.33998v1

Accurate streaming detection of action changes in video sequences

Groupwise Selective State-Space Filtering for Accurate and Streaming Action Boundary Detection

Abstract: Action boundary detection partitions untrimmed video into intervals without assigning action classes. We present a boundary-detection adapter operating on pre-extracted video features, learning temporal representations via groupwise selective scans. Learned group fusion and temporal modeling convert these into transition scores, which are decoded into boundary timestamps. Trained with boundary-time supervision, the class-agnostic model is evaluated on Breakfast, GTEA, and 50Salads using temporal tolerances and bipartite matching, achieving boundary $F_1$ scores of 0.457, 0.622, and 0.611. A stateful variant enables feature-streaming inference with zero neural look-ahead, one-sample peak confirmation, and bounded memory. Downstream systems can subsequently assign s

Sun 27 SeptComputer Vision and Pattern Recognition
The gist
Detecting where one action ends and another begins in long videos is important for analyzing activities without labeling what the actions are. The authors describe a method that looks at video features over time in groups to find these boundaries accurately. Their approach works without knowing the action types and can handle live video streams with little delay and limited memory. It was tested on different datasets and improved the detection of action transitions.
Open → 2609.33400v1

Omnimodal agent improves referring video segmentation with precise reasoning

OPERA: A Unified Omnimodal Progressive Spatio-Temporal Reasoning Agent for Referring Video Segmentation

Abstract: Referring video segmentation with heterogeneous multimodal queries---spanning text, audio, and reference images---demands both robust cross-modal understanding and precise spatio-temporal reasoning. We propose OPERA (Omnimodal Progressive spatio-tEmporal Reasoning Agent), a unified reasoning agent built on a single MLLM that performs dual-axis progressive reasoning via three specialized stages. Along the temporal axis, a Temporal Reasoning Agent narrows the frame search space through coarse-to-fine filtering to identify the most informative key frame. Along the spatial axis, a Distillation Agent establishes what to locate via cross-modal semantic distillation, and a Grounding Agent enhanced with GRPO determines where the target appears, with dense mask propagation completing the pixel-level output. OPERA sets a new state of the art on OmniAVS and Ref-AVS and transfers zero-shot to standard referring video segmentation benchmarks.

Sun 27 SeptComputer Vision and Pattern Recognition
The gist
Referring video segmentation means identifying specific things or moments in a video based on clues like words, sounds, or pictures. The authors created OPERA, a system that carefully looks through video frames over time and uses different types of clues to find exactly what to highlight in the video. OPERA filters information step-by-step to focus on important frames and spots, producing detailed masks that outline the target. This new system works better than previous ones on several video segmentation tests.
Open → 2609.33338v1

Semi supervised visible infrared object detection with aligned consensus teacher

Aligned Consensus Teaching for Label-Efficient Oriented Object Detection in Weakly-Aligned Visible-Infrared Imagery

Abstract: Visible-infrared object detection (VIOD) detects objects with oriented bounding boxes from paired visible and infrared images. Existing methods depend on costly dual-modality annotations. Semi-supervised learning can reduce this burden, but extending it from single-modal detection to VIOD is challenging. In the practical image-pair-level setting considered here, only a few pairs are labeled in both modalities, while the rest are completely unlabeled. This limited supervision creates three challenges: (i) too few labeled boxes for robust cross-modal alignment; (ii) pseudo-label errors caused by branch-wise misses accumulate during self-training; and (iii) tail-class annotations become critically scarce as the labeling budget decreases. We propose Aligned Consensus Teacher (ACT) for label-efficient VIOD in this setting. Its Cycle-Consistent Region Alignment (CRA) combines cycle consistency and sparse anchors with reliability-weighted regional matching. Cross-Modal Consensus Mean-Teacher (CMC-MT) forms consensus pseudo labels under pair-preserving views to recover branch-wise misses and supervise unlabeled pairs. Text-Guided Cross-Modal Instance Augmentation (TG-CMIA) uses a vision-language scene prior to compose tail-class instance pairs while preserving RGB--IR offsets. To the best of our knowledge, ACT is the first framework to study semi-supervised VIOD under this image-pair-level setting. Experiments on DroneVehicle and VEDAI show consistent gains across annotation ratios. With 10\% labeled pairs on DroneVehicle, ACT reaches 94.3\% of the mAP obtained by the same detector under full supervision. Code and models will be available on GitHub to facilitate future work.

Wed 16 SeptComputer Vision and Pattern Recognition
The gist
Detecting objects in images taken by both regular cameras and infrared cameras usually needs lots of detailed labels, which are expensive to get. The authors developed a method called Aligned Consensus Teacher that uses only a small amount of labeled image pairs and many unlabeled pairs to learn accurately. Their approach smartly aligns objects between visible and infrared images and improves predictions by combining knowledge from both types of images. This method works well even when only a small fraction of the image pairs are labeled, achieving performance close to fully labeled training.
Open → 2609.18124v1