TGRIP: A Text-Guided Approach to Vehicle Instance Prediction in Autonomous Driving
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors propose a new method called TGRIP to improve how self-driving cars predict the future positions of objects around them. Unlike previous methods that treat all moving things the same, their approach uses language-based models to add detailed descriptions of what the objects are, helping the system understand scenes better. They train their model with these semantic hints alongside traditional geometry information, allowing it to make more accurate predictions especially in tricky situations like intersections. Their tests showed better results than earlier methods on a standard dataset.
Bird's-Eye View (BEV)Instance predictionAutonomous drivingSemantic segmentationVision-language modelsSpatio-temporal representationOccupancy regressionOptical flowTeacher-student pipelinenuScenes dataset
Authors
Miguel Antunes-García, Santiago Montiel-Marín, Fabio Sánchez-García, Rodrigo Gutiérrez-Moreno, Rafael Barea, Luis M. Bergasa
Abstract
Bird's-Eye View (BEV) end-to-end instance prediction has emerged as a robust paradigm for autonomous driving perception, effectively mitigating the error propagation inherent in traditional modular pipelines. However, current state-of-the-art approaches rely predominantly on geometric supervision, such as occupancy regression and optical flow, effectively treating scene agents as generic moving obstacles. This absence of explicit semantic awareness imposes limitations on the capacity of the model to solve ambiguities in complex scenarios, particularly those where object-specific behavior is essential for accurate forecasting (e.g. overtaking, intersections). In this paper, we introduce Text-Guided Representation for Instance Prediction (TGRIP), a novel framework that bridges this gap by injecting rich semantic priors into the instance prediction loop. The proposed teacher-student pipeline employs Vision-Language Foundation Models to generate dense, semantic-enhanced BEV maps from multi-camera images. These maps serve as auxiliary supervision during training, guiding the network to learn spatio-temporal representations that are not only geometrically consistent but also semantically discriminative. To the best of our knowledge, this represents the first attempt to unify semantic guidance with the temporal task of future instance prediction. The experimental results demonstrate that TGRIP surpasses existing state-of-the-art models in nuScenes, validating the hypothesis that semantic enrichment is a fundamental element for robust, end-to-end motion prediction. Code is available on https://github.com/miguelag99/TGRIP.