Sensor fusion bird’s eye view improves remote control of full-size autonomous cars
Revisiting Direct Control: A Field Evaluation of Sensor Fusion Efficacy in Teleoperating Full-Scale Autonomous Vehicles via Bird's Eye View
RoboticsHuman-Computer Interaction
Summary
Remotely controlling self-driving cars is important when the car’s automatic systems fail. The authors tested three different ways to show the driver the surroundings: a camera view, a bird’s-eye view made from LiDAR, and a bird’s-eye view that combined camera and LiDAR data. They found that the combined view helped drivers complete tasks faster and with less mental effort than the other views. This suggests that mixing different sensor data creates safer and easier ways to remotely drive big self-driving cars.
What this means in practice
- •For autonomous vehicle operators: Use sensor-fused bird’s-eye views to improve control accuracy and reduce workload during remote driving interventions.
- •For industrial vehicle fleet managers: Implement sensor fusion interfaces in teleoperation setups to enhance safety margins and operator effectiveness for remote vehicle control.
Authors
Stanisław Kuczma, Sławomir K. Tadeja, Michał R. Nowicki
Abstract
In autonomous driving, robot teleoperation serves as a critical safety fallback mechanism, requiring operators to assume direct control in the event of system failure. While recent research has pivoted toward high-level supervisory control or simulation-based studies, these approaches often fail to capture the physical dynamics and sensory demands of driving a full-scale vehicle. To address this gap, we evaluated the impact of spatial data representation on operator performance using a real car-sized vehicle in a field experiment with $N = 18$ participants. We compared three visualization interfaces: (i) a standard camera feed baseline, with two bird's-eye views (BEV) from (ii) a raw LiDAR point cloud, and (iii) a sensor fusion of LiDAR and camera data streams. Contrary to prior passive observation studies suggesting that sensor-fusion BEV may hinder awareness, our active-control results demonstrate that such representation significantly outperforms the standard LiDAR BEV in perceived usability ($p \leq 0.0001$) and cognitive task load ($p = 0.01$, NASA-TLX score: 34.58/100) while also being significantly better than the Camera-only view ($p = 0.043$). The fused interface also led to statistically significant reductions in task completion time ($p = 0.0461$) and superior usability (SUS: 77.9/100) compared to those of the Camera-only and LiDAR-only views. By synthesizing objective performance data with comprehensive subjective assessments, we demonstrate that BEV fusion maximizes safety margins while minimizing the operator's mental demand. These findings challenge existing teleoperation conventions and provide an empirically validated approach for designing robust remote piloting systems.