World2vec learns compact accurate models for understanding environments
world2vec: Mechanistic world models through latent dynamics learning
Machine Learning
Summary
Understanding how things change and behave in complex environments is hard, especially when you only see images and actions without clear instructions. The authors created world2vec, a system that learns to predict future states of the environment accurately using fewer resources than other methods. This helps in better controlling systems and discovering how they work internally, which is valuable for scientific and engineering purposes. The approach works well even over long time periods and across many types of systems.
What this means in practice
- •For robotics engineers: Generate compact and accurate models from visual inputs to improve robot control and prediction in continuous environments.
- •For industrial automation teams: Use learned mechanistic models to identify system dynamics and enhance process monitoring and adjustment without relying on reward signals.
Authors
Tobias Schmidt, Rodrigo Gonzalez Laiz, Krisztián Szuppinger, Steffen Schneider
Abstract
World models can achieve strong control performance without faithfully capturing the dynamics of their environment, limiting their potential for mechanistic interpretability and scientific discovery. Here, we introduce world2vec to learn mechanistic world models without reward information directly from observations and actions. World2vec learns latent dynamics that faithfully reproduce the true system over long prediction horizons, with at least one order of magnitude fewer parameters than DreamerV3, TD-MPC2 and LeWM. This enables state of the art performance in controlling continuous environments from pixels and accurate system identification across a benchmark of 71 attractors, demonstrating how world models are amenable to scientific discovery.