Language based trajectory prediction improved by numerical expert feedback
Revisiting Numerical Forecasting Models for Language-Based Trajectory Prediction
Computer Vision and Pattern Recognition
Summary
Predicting where people will walk is hard because it involves understanding both physical motion and social behavior. The authors revisit a way to represent paths using language models but add numerical knowledge from specialized predictors to better guide predictions. Their system, MoRE, uses rewards from multiple numerical experts to refine a language-based predictor, improving accuracy and reducing collisions in pedestrian crowd simulations. This approach combines the strengths of language context and precise coordinate feedback without slowing down the predictions.
What this means in practice
- •For autonomous vehicle engineers: Improve pedestrian path prediction by integrating social and motion cues using a combined language and numerical expert system.
- •For video game developers: Enhance realistic crowd movement by leveraging improved trajectory prediction with language models refined by motion experts.
Authors
JunGyu Lee, Inhwan Bae, Hae-Gon Jeon
Abstract
Language-based trajectory predictors represent coordinates as discrete tokens and learn auxiliary tasks such as destination and group reasoning. This formulation enables the model to capture behavioral intent and social context beyond coordinate dynamics alone. However, token-level objectives provide only indirect guidance for continuous coordinate-space dynamics. To address this limitation, we introduce MoRE (Mixture of Reward Experts), a refinement framework that transfers numerical forecasting priors into a pretrained language-based predictor through reinforcement learning. Five frozen numerical predictors provide complementary coordinate-level knowledge of motion and interactions. Their predictions are converted into expert rewards and combined through an uncertainty-weighted consensus that penalizes disagreement. A ground-truth reward anchors the prediction to the target trajectory. To focus refinement on difficult cases, MoRE refines the policy using the top 1% of training samples ranked by predictive entropy. Expert predictions are computed once and cached before PPO training, so the experts are not run during policy updates or inference. In this way, MoRE combines the contextual modeling of the language-based predictor with coordinate-level feedback from numerical experts. On ETH-UCY, MoRE reduces ADE from 0.22 to 0.20 m and FDE from 0.32 to 0.29 m. Relative to the base policy, ADE decreases by 17.9% on SDD and 12.7% on NBA. On ETH-UCY, MoRE also reduces collision rates and better matches ground-truth pedestrian spacing, without increasing measured inference memory or latency. The project page is available at https://jungyu0413.github.io/MoRE/.