Towards Collaborative Joint Perception and Prediction: Framework, Baseline Evaluation, and Deployment Perspectives
2026-08-10 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study how connected self-driving cars can better understand and predict the actions of other road users by sharing sensor information. They propose a new approach called Collaborative Joint Perception and Prediction (Co-P&P) that combines seeing (perception) and guessing future movements (prediction) to reduce errors and handle blocked views. Their experiments show that merging information at the detection or tracking stages works better than merging after prediction. They also created a prototype system that shares compressed sensor data and improves prediction accuracy while using much less communication bandwidth.
Connected Autonomous VehiclesV2X communicationCollaborative PerceptionMotion PredictionSensor FusionPoint CloudsNeural CompressionFutureDetDetection-level FusionTracking-level Fusion
Authors
Lei Wan, Hannan Ejaz Keen, Alexey Vinel
Abstract
Connected Autonomous Vehicles (CAVs) increasingly exploit Vehicle-to-Everything (V2X) communication to exchange multi-source sensor information, enabling advanced Collaborative Perception (CP) capabilities. Extending beyond these capabilities, this work focuses on Collaborative Joint Perception and Prediction (Co-P&P), a paradigm that unifies CP with motion prediction to mitigate two persistent challenges: the accumulation of perception errors and visual occlusions. We present a conceptual framework for Collaborative Joint Perception and Prediction (Co-P&P) that improves motion prediction of surrounding road users, thereby enhancing situational awareness in complex and dynamic traffic environments. Building upon our preliminary study, this extended version compares the performance of different fusion strategies and establishes baseline performance for a modular design of perception and prediction. Experimental results show that prediction-level fusion leads to a decline in overall system performance compared to detection-level or tracking-level fusion. We further implement a minimal end-to-end Co-P&P prototype that couples collaborative point-cloud sharing via the RENO neural codec with joint detection-forecasting via FutureDet, showing that collaboration improves forecasting accuracy while neural compression preserves this benefit at roughly 34x lower communication bandwidth.