Object relative representations boost reinforcement learning in games
Reframing Object-Centric Reinforcement Learning: From Absolute to Relative Representations
Machine LearningArtificial Intelligence
Summary
Many AI agents decide what to do based on where objects are arranged, but they often use a fixed map that doesn't change if the whole scene moves around. This means they have to learn the same behavior separately for every possible location. The authors suggest describing objects relative to key 'anchor' objects, so the AI learns behaviors that work even when scenes shift or rotate. They created a new AI system called ORBiT that uses these relative views and found it works better in many video games, even learning to play new games better than some existing methods.
What this means in practice
- •For game ai developers: Build game-playing agents that learn better by understanding objects relative to each other, improving performance and generalization to new levels.
- •For robot navigation engineers: Create robots that can understand and navigate environments more reliably by using relative coordinates tied to key objects instead of fixed global maps.
Authors
Cedric Derstroff, Jannis Blüml, Janosch Moos, Michael Girstl, Zihan Ye, Debora Clever, Kristian Kersting
Abstract
Object-centric reinforcement learning agents typically receive each object's position in a fixed global frame, even though their decisions usually depend on where objects are relative to one another. These absolute views often fail to capture the local context that is most relevant to the agent's behavior. As a result, an agent must relearn the same response at every location where a configuration can occur, and its performance collapses when the scene is merely shifted. In this work, we propose a shift toward Object-Relative Coordinates fixed to Anchor (ORCA), where entities are described in a relative frame of reference. To operationalize this idea, we introduce Object-Relative inductive-Bias Transformers (ORBiT), a transformer-based architecture for processing anchor-relative inputs in reinforcement learning. Our results suggest these relative representations improve performance for both MLP and transformer agents. Furthermore, we demonstrate that ORBiT outperforms the pixel-based PPO baseline, achieving an aggregate HNS IQM greater than 1.0 over 13 Atari games. ORBiT learns the game of Foosball from scratch, dominating all baselines. In our controlled Tag environment, we further show that ORCA extends to rotations, that ORBiT generalizes to unseen numbers of distractors, and that these gains require a decision-relevant anchor.