Orbis 2: A Hierarchical World Model for Driving
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine LearningRobotics
Summary
The authors created a driving world model that works on two levels: a high-level part that predicts the general scene far into the future, and a low-level part that fills in detailed visuals based on that. This design helps the model understand both the big picture and detailed changes in driving scenes. They also use a new two-step training method that first teaches the model to make diverse guesses and then to make steady precise predictions, improving how the model learns and performs. Their method outperforms existing models in predicting driving scenes over long times and responding to driving changes.
Authors
Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso, Karim Farid, Jonannes Dienert, Rajat Sahay, Thomas Brox
Abstract
Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/