SetOPD improves multimodal detection from few visual examples
SetOPD: From Few Visual Exemplars to Multimodal Candidate Sets for Remote-Sensing Open-Prompt Detection
Computer Vision and Pattern Recognition
Summary
Detecting objects in satellite or aerial images can be tricky, especially when you only have a few example pictures to work from. The authors found that many current methods combine all example images into one summary, which can lose important details. They created a new way called SetOPD that keeps each example separate and lets the system compare text and images more effectively. This helps the detector better understand and find targets by carefully mixing information from both descriptions and multiple example images.
What this means in practice
- •For remote sensing analysts: Detect targets in aerial images more accurately when only a few example images and textual descriptions are available.
- •For security monitoring teams: Improve surveillance systems that rely on combining small sets of visual and textual evidence to spot unusual or targeted objects remotely.
Authors
Jinlong Hu, Yi Zhang, Zhiqi Xia, Yikang Zhou, Shunping Ji
Abstract
Open-prompt detectors allow users to specify targets with text, visual exemplars, or both. We argue that existing designs underuse the visual modality in two ways. First, multiple exemplars are commonly compressed into a single class-level embedding. This textualizes visual prompting: the resulting vector plays the role of another class name, is often aligned to or injected into the text pathway, and may be suboptimal when only a few heterogeneous exemplars are available. Second, existing methods interact primarily in prompt or representation space, before modality-specific detection states are formed. We address both issues from a set perspective: \setopd preserves modality-specific decoding states from a shared prompt-conditioned initialization and performs explicit multimodal collaboration at the candidate-state level. For the first issue, we introduce \br prompting, which reads every boxed exemplar in its full scene context and decomposes the pooled support evidence into a Base anchor and a learned Residual correction; the resulting prompt has fixed capacity regardless of the number of examples and drives its own visual detection pathway. For the second, we recast text--visual collaboration from representation-level fusion into a candidate-set modeling problem. Paired-Query Arbitration (\pqa) then performs explicit cross-modal state arbitration only after modality-specific candidate states have been formed. The two readers share query initialization so their candidates are paired by index; a learned gate arbitrates within each pair, followed by a permutation-equivariant module that reasons over the fused set.