Papers for

surveillance operators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Streaming video model starts conversations without prompts

AURA 2.0: From User-Driven Responses to Model-Initiated Conversations in Streaming Video

Abstract: Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.

Thu 12 NovComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Most AI helpers wait to be asked questions before answering, missing chances to talk when something interesting happens on screen. The authors designed AURA 2.0, a system that can notice important events in videos and start conversations on its own. It can also use extra tools and clearly link its answers to objects shown in the video. This helps the system keep track of long videos and chat more naturally over time. The authors tested their system and made the code available for others to use.
Open → 2611.17048v1

Large vision language models detect human-like event boundaries in videos

Can an LVLM Perceive Event Boundaries Like a Human?

Abstract: Large Vision-Language Models (LVLMs) reason effectively about video content, but whether they perceive its event boundaries in a manner consistent with humans remains unclear. We first investigate this through Generic Event Boundary Detection (GEBD), which identifies event boundaries naturally perceived by humans in continuous video, without relying on predefined event categories. On Kinetics-GEBD dataset, LVLMs miss many boundaries when predicting directly from frames, but effectively refine candidates proposed by an unsupervised discovery method, achieving state-of-the-art unsupervised GEBD performance. We extend this discovery-refinement approach to hour-long egocentric video, where boundary discovery requires a broader semantic scale and whole-video LVLM refinement becomes impractical. We address these challenges through coarse-to-fine temporal discovery and localized LVLM accept/reject judgments. Human evaluation shows that the LVLM's agreement with group consensus falls within the range observed among human annotators, with disagreements concentrated on low-consensus cases. Using this pipeline, we construct EgoBoundary, a benchmark for event boundary detection in hour-scale egocentric video. Representative supervised and unsupervised methods degrade substantially on EgoBoundary, highlighting the challenge of long-horizon event boundary detection.

Thu 12 NovComputer Vision and Pattern Recognition
The gist
Figuring out where one event ends and another begins in a video is something humans do naturally. The authors studied whether large vision-language models (LVLMs) can spot these event boundaries like people do. They found that while these models miss many boundaries when working directly with video frames, they do well when refining candidate boundaries suggested by an unsupervised method. This approach works even for very long videos from a first-person perspective. They also created a new benchmark dataset called EgoBoundary to test event detection on long videos, where existing methods struggle more.
Open → 2611.16452v1

BiHashFormer improves object detection in high-resolution wide images

BiHashFormer: Hash-Driven Dual-Branch Transformer for Efficient Object Detection in HRW Shots

Abstract: Recent advances in gigapixel-level imaging have brought High-Resolution Wide shots to the forefront of research. However, these images present significant challenges: extreme sparsity of foreground, gigapixel-level resolutions and interleaving of foreground and background. This causes traditional detectors and attention mechanisms to be hindered by background, resulting in inefficiency and inaccuracy. To tackle this problem, we propose BiHashFormer, a dual sparsified branch transformer built on hashing. RoI Selector will predict the foreground proportion for top-k selecting target-containing windows. The compression branch processes windows selected and refines attention into HashAttention by discarding half of the key-value pairs, reducing cost and improving efficiency. The compensation branch uses HashMiner to perform low-cost hash searches in the remaining regions, recovering features of missed objects. Experiments confirm that the selection relationship in HashAttention is one-way and reveal the preference of different queries for keys. In experiments on the gigapixel benchmark PANDA, BiHashFormer reduces 37.5% backbone FLOPs while improving AP50 to 81.0%. Moreover, this parallel dual-branch structure allows the model to avoid inference delays caused by inter-branch dependencies, while achieving much greater stability than single-branch models, specially with low rates of RoI selection.

Thu 29 OctComputer Vision and Pattern Recognition
The gist
Detecting objects in extremely large, detailed images is hard because objects are sparse and surrounded by lots of background. The authors introduce BiHashFormer, a method that uses two parallel parts working together to focus on important areas and reduce unnecessary processing. One part zooms in on likely areas with objects, while the other checks the rest efficiently to recover any missed spots. This approach makes detection faster and more accurate on huge images, like gigapixel photos.
Open → 2610.39937v1

Moving regions identified precisely under camera motion in videos

Moving Regions, Not Moving Objects: Segmenting What Actually Moves under Camera Motion

Abstract: Moving object segmentation is conventionally formulated at the sequence and object level, marking an entire instance as moving once it moves anywhere in a video. Recent instantaneous formulations distinguish moving from stationary intervals, but still generate masks at the object level through frozen segmentation foundation models. Because the mask is derived from an instance-level proposal, motion is typically attributed to the entire instance rather than localized to the regions that actually move. Moreover, such instance-level predictions can produce masks even when no part of the instance is actually moving, revealing a more fundamental issue: motion is not an intrinsic property of an instance; it can only be determined by comparing at least two frames. Building on this premise, we introduce MoRSe, which jointly interprets two frames to predict a moving-region mask, distinguishing actual scene motion from apparent motion induced by the camera. A single RGB encoder replaces multi-stage foundation stacks, so no optical flow, point tracks, or separate mask generator is needed at inference. We release MoRSe-5K, a benchmark of 42 sequences with Static, Dynamic-Static, and Dynamic settings designed to evaluate the model's ability to distinguish and localize moving and stationary regions under different camera and scene motion conditions. MoRSe outperforms both specialized moving object segmentation methods and dense 4D reconstruction methods with motion-thresholded predictions, while producing the fewest false positives on sequences where nothing moves.

Thu 22 OctComputer Vision and Pattern Recognition
The gist
When a camera moves, it’s hard to tell which parts of a video are really moving objects versus just changes caused by the camera itself. The authors developed MoRSe, a method that compares two video frames together to find the exact moving parts, not just whole objects that might only partially move. This method avoids common mistakes and works well even when nothing in the scene moves. They also created a benchmark dataset called MoRSe-5K to test these abilities under various motion conditions.
Open → 2610.31209v1

Atmospheric turbulence effects reduced using triple neural networks

TriINR: Unsupervised Atmospheric Turbulence Mitigation with Triple Implicit Neural Representations

Abstract: Atmospheric turbulence degrades image quality in long-range imaging. The degradation is generally modeled with two spatio-temporally varying components, tilt and blur. Recent unsupervised turbulence mitigation methods have demonstrated the effectiveness of representing a sharp image as an implicit neural representation (INR). However, their tilt and blur representations remain frame-dependent, and the number of optimization parameters grows with the number of input frames. To address this limitation, we propose TriINR, an unsupervised turbulence mitigation method that recovers a sharp image from multiple degraded frames. TriINR represents the sharp image, tilt, and blur as three implicit neural representations. By modeling tilt and blur as spatio-temporal INRs, TriINR achieves frame-count-independent degradation representation and memory-efficient optimization. We further incorporate physical and statistical properties of atmospheric turbulence as constraints on the INRs to improve restoration accuracy. Experiments on synthesized turbulence data with three degradation strengths show that TriINR restores images more robustly across varying turbulence strengths than prior TM methods and achieves the highest accuracy among the evaluated unsupervised methods. Our method also demonstrates robustness on real long-range scenes containing additional non-ideal motion.

Fri 16 OctComputer Vision and Pattern Recognition
The gist
Long-distance images often look blurry or wobbly because of atmospheric turbulence, which shifts and distorts them over time. The authors propose TriINR, a method that uses three connected neural networks to learn and separate the sharp image, the wobbly shifts (tilt), and the blurring all at once. This approach does not depend on how many blurry frames it gets, making it more memory-efficient and accurate. They also add knowledge about how turbulence behaves to help the networks restore clearer images. Tests show TriINR works better than earlier methods, even on real-world scenes with unexpected movements.
Open → 2610.23454v1

Cross-view object recognition improves using 3d visibility aware training

3D-VITAL: Visibility-aware Identity Training from Artificial Liftings of 3D Reconstruction model for Cross-View Object Re-Identification

Abstract: We address the Cross-View Object Re-Identification (ReID) task, aiming to match identities across drastically different viewpoints. We focus on Single-View Aerial-Ground ReID (SV AG-ReID), where models train on one viewpoint and must generalize to another at inference. To bridge this viewpoint gap, related work employs generative models to synthesize novel views from source images as additional training data. We leverage recent advances in large-scale single-view 3D mesh reconstruction to lift source images into textured meshes and to render them from new viewpoints. However, this pipeline produces unreliable data in two distinct ways. First, the reconstruction may fail and produce geometrically incorrect meshes, resulting in severe corruption in rendered novel views. Second, even when the geometry is correct, the rendered novel views contain hallucinated textures in regions not visible in the source image. The reconstructed mesh carries a per-pixel partition between regions visible from the source view and regions that the reconstruction model had to hallucinate, available as a deterministic byproduct of rendering. We introduce this per-pixel visibility map and exploit it in a two-stage design. Cross-View Consistency Filtering (CVCF) discards geometrically incorrect meshes by ensuring multi-view consistency between source and novel views over visible patches. Visibility-Aware Attention Supervision (VAAS) leverages the visibility map to guide the model's attention toward reliable regions of the rendered view during training. Combined, these two components enable our method, 3D-VITAL, to better leverage the synthetic signal from single-view 3D mesh reconstruction. 3D-VITAL sets a new state-of-the-art on three SV AG-ReID benchmarks. Code and data will be released at https://github.com/TurtleSmoke/3D-VITAL.

Mon 12 OctComputer Vision and Pattern Recognition
The gist
Matching the identity of objects seen from very different viewpoints, like from the ground and from the air, is hard. The authors use 3D models created from single images to generate new views and help computers learn better. They found that these 3D models sometimes contain mistakes or fake details, so they created methods to ignore bad parts and focus on trustworthy areas during training. This approach improves accuracy for recognizing objects across views.
Open → 2610.16692v1

OneStreamer improves real-time video memory and responses with proactive captions

OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction

Abstract: Streaming video LLMs must retain evidence before its relevance to future tasks is known and respond when sufficient evidence becomes available. The challenge is to form reusable factual memory without compromising real-time perception. We introduce OneStreamer, which jointly learns query-independent evidence recording and task response through a shared proactive generation process. Its Proactive Hierarchical Caption Memory (PHCM) produces time-grounded local-detail captions and summaries of completed events. Streaming caption targets supervise the interpretation of observed video prefixes during training. At inference, model-generated records complement a recent visual window, providing reusable factual context without revisiting historical visual features. Proactive State Transition Learning (PSTL) reduces the dominance of repeated waiting states by preserving supervision at all output anchors and selecting representative state-change and state-persistence tokens. We further develop a streaming data synthesis pipeline that aligns output content and timing with available evidence. Combining the resulting streaming captions and QA with cleaned open-source data yields OneStreamer-1M, a broad-coverage streaming video interaction dataset with over one million records spanning diverse tasks. Our 4B model achieves the best results among the compared methods across all eight evaluated streaming video understanding benchmarks. Ablations show that retaining generated captions improves historical QA without degrading real-time perception. PSTL also outperforms dense state supervision while supervising only 27.5% of annotated state tokens. Together, these results support proactive generation as a shared learning interface connecting perception, memory formation, and timely response in streaming video interaction.

Thu 1 OctComputer Vision and Pattern Recognition
The gist
Understanding videos as they stream in real time is hard because it’s unclear what information will be useful later. The authors created OneStreamer, a system that writes down key facts and summaries as it watches, so it remembers important details without rewatching. It learns when to write and respond, improving how well it understands ongoing events and answers questions. They built a large dataset to train and test their system, showing it works better than previous methods.
Open → 2610.01762v1

Sprout builds and updates video memory dynamically during questioning

Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding

Abstract: Long video understanding relies on video memory to overcome the context limits of multimodal large language models. Existing methods follow a build-then-reasoning pipeline: memory is built offline for the entire video, then reasoned over as a static source. In practice a long video is shared by several questions, and this pipeline is costly at both ends: with few questions, building memory for the whole video costs far more than answering them; with many questions, the memory is never updated, so what is learned while answering questions is lost to the next question. To alleviate these, we introduce Sprout, an agentic framework that builds memory while reasoning: a temporal tree that sprouts detailed nodes as questions are answered. The agent watches the video segment by segment at a low frame rate, stopping when the current question can be answered, remembers each segment as a coarse node of the tree, and revisits key intervals at a higher frame rate to refine the tree with the recovered details. Once a segment is recorded as text, its video input is removed from the context history, while the original video remains reachable through the video tools. The memory tree and prior question--answer records persist across questions, so the memory is online and dynamic: built from the first question onward and updated by every question thereafter. We find that replacing accumulated video inputs with textual memory substantially reduces context usage while maintaining accuracy, with slight improvements in some settings. Across benchmarks on three models, Sprout achieves competitive or improved accuracy relative to representative offline memory methods, with no upfront construction stage and lower context cost per question.

Mon 28 SeptComputer Vision and Pattern Recognition
The gist
Long videos are hard for AI to understand all at once because they contain too much information. Existing methods first build a memory of the whole video and then try to answer questions, which is slow and wastes effort. The authors created Sprout, a system that watches videos in chunks and builds a memory while answering questions, saving time and updating what it knows as new questions come. Sprout stores video details as a tree of text summaries, keeping memory small and allowing the AI to revisit important parts more deeply when needed. This approach performs as well as or better than earlier methods but with less cost and no big setup step.
Open → 2609.35497v1

Streaming video agent selects key frames without future knowledge

SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selection

Abstract: Most keyframe selection studies focus on offline settings, assuming access to the full video and query in advance. In contrast, real-world streaming scenarios require online frame selection under unknown video duration, without access to either the query or future frames during selection. To address this, we introduce Streaming Video Memory (SVMem), a compact and representative memory of previously observed content, updated continuously as the video stream unfolds. Building on this setting, we propose the Streaming Video Memory Agent (SVMemAgent), which dynamically maintains a memory by deciding at each timestep whether to replace an existing memory frame with the incoming frame or discard it. SVMemAgent is trained using Group Relative Policy Optimization (GRPO) with task-driven rewards derived from diverse question-answer pairs, implicitly exposing the policy to a distribution of queries during training so that SVMem retains generally informative frames at inference, when queries are unavailable. Experiments on both online and offline video benchmarks show that SVMemAgent consistently outperforms online frame selection baselines and achieves competitive performance with offline methods that assume access to the full video and query. Through task-driven rewards, SVMemAgent learns an emergent keyframe selection policy that prefers frames containing textual information, which may benefit downstream VideoQA tasks.

Wed 16 SeptComputer Vision and Pattern Recognition
The gist
Selecting important frames from videos usually requires seeing the whole video first. The authors studied how to pick key frames from a video as it streams, without knowing how long it is or what questions might be asked about it. They built a system called SVMemAgent that keeps a small memory of important past frames and decides whether to update it with new frames on the fly. Their system learns from many question-answer examples so it can keep generally useful frames even when no question is known. This helps in tasks like video question answering by focusing on frames with important information.
Open → 2609.18540v1