Vision guided control speeds up autonomous racing with onboard cameras
VIGOR-MPC: Vision-Guided Objective Regulation MPC
Robotics
Summary
Controlling a robot or car using only what its camera sees is hard, because the view can be unclear and the computer has to work fast. The authors created a way to teach a simple controller to copy a complicated one that knows everything, by learning from what the camera sees. This method helps the simple controller make quick decisions that almost match the complex controller’s performance, using much less computing power. They showed it works well in race track settings, even on new tracks the system has not seen before.
What this means in practice
- •For autonomous vehicle engineers: Improve real-time control of self-driving vehicles using only onboard cameras with lightweight predictive models.
- •For robotics hardware developers: Design robots that balance onboard perception and control computation to run faster control loops with limited hardware.
- •For drone racing teams: Enable racing drones to better navigate tracks using vision-based control strategies that reduce onboard processing while maintaining performance.$Commercial implications: This paper enables racing drone manufacturers to sell improved autonomous navigation systems combining vision and control with efficient computation.
Authors
Jelena Trisovic, Rahel Rickenbach, Marek Landert, Panagiotis Minos, Melanie N. Zeilinger
Abstract
Controlling an agent in an environment solely perceived through onboard vision is challenging: perception yields noisy, variable-length, and partially occluded context, while onboard compute is shared between perception and control, limiting the planning horizon executable in real time. We propose VIGOR-MPC, a modular student-teacher framework, agnostic to the perception pipeline, which addresses reference tracking across navigation tasks by distilling a privileged, field-of-view-aware long-horizon MPC into a computationally lightweight short-horizon MPC whose cost parameters are determined online from vision. The teacher operates in simulation with full state knowledge and a multi-trajectory formulation that guarantees the agent can stay within its field of view. The student observes only the noisy, occluded context produced by a perception module and learns, via a multi-level recurrent mapping, to infer cost parameters that aim to recover the teacher's behavior despite missing information. We evaluate VIGOR-MPC on vision-based autonomous racing in simulation and on hardware. Using only onboard perception of the track, VIGOR-MPC generalizes to tracks unseen during training and approaches the lap times of the privileged long-horizon teacher while reducing per-step computation by over 90%.