Summary
Detecting objects in 3D space using data from multiple vehicles and roadside sensors is challenging because they produce different, sometimes conflicting information. The authors propose VeriFuse, a system that first gathers proposals from each agent, then uses a vision-language model to pick the best candidate or improve existing ones, while rejecting unreliable detections. This combination helps make better decisions about what objects are around, improving overall detection accuracy without heavy computation. Their tests show that VeriFuse keeps vehicle detection quality high even with some delay in communication.
What this means in practice
- •For autonomous vehicle engineers: Integrate VeriFuse to better combine multiple 3D detection sources from vehicles and roadside units, improving accuracy and robustness of environmental perception.
- •For urban traffic control teams: Use VeriFuse to refine roadside sensor data by leveraging cross-source semantic analysis for more reliable vehicle detection in busy environments.
Tested on one dataset.
Authors
Hongyi Lin, Yiyao Liu, Qi Kang, Heye Huang, Yang Liu, Haris Koutsopoulos, Jinhua Zhao
Abstract
Vision-language models (VLMs) have demonstrated strong scene understanding and semantic judgment across diverse tasks, but their appropriate role in cooperative perception remains unclear. Directly asking a VLM to regress 3D detections is unreliable and computationally expensive, whereas using it to select the output of a single source discards useful information from other agents. We introduce VeriFuse, a bounded arbitration framework for vehicle-infrastructure cooperative 3D detection. Each agent first produces detections independently. Around each vehicle and roadside proposal, VeriFuse generates source-conditioned geometric candidates and combines the original detections, their perturbations, and cross-source hypotheses into a unified candidate pool. A frozen VLM then chooses among three admissible actions: SELECT an adequate candidate; REFINE an existing anchor when an object is supported but all candidates are geometrically inadequate; or REJECT an unsupported infrastructure-only proposal. Experiments on the DAIR-V2X dataset show that VeriFuse achieves 0.494/0.357 cooperative 3D AP50/AP70 and limits the relative vehicle-side BEV AP50 drop under a 300 ms delay to 1.7%. Overall, VeriFuse assigns the VLM a clear and constrained role in cooperative perception: semantic reasoning resolves ambiguity among cross-agent hypotheses, while deterministic constraints determine the final 3D geometry.