Reinforcement learning improves by using local state symmetries
Groupoid-Based Internal State Representations for Reinforcement Learning with Local Symmetries
Machine LearningArtificial Intelligence
Summary
Reinforcement learning helps computers learn by trying actions and seeing results, but usually assumes the rules apply everywhere the same way. This paper shows that by using a mathematical tool called groupoids, computers can recognize when parts of a problem are similar only in certain places, not everywhere. This helps the computer learn faster and better in complex situations where rules change depending on the context. The authors tested their idea and found it works better than standard methods in big, complicated environments.
What this means in practice
- •For robotics engineers: Improve robot learning by recognizing context-specific similarities in environments to speed up training in complex tasks.
- •For game developers: Design AI that learns faster in games with parts of the world showing recurring but local patterns.
Authors
Ben Opperman, Eduardo Alonso, Esther Mondragón
Abstract
Symmetries play a central role in reducing the complexity of reinforcement learning problems, yet most existing approaches rely on fixed group actions or predefined state abstractions. Classical reinforcement learning algorithms typically assume a globally structured Markov decision process with uniformly applicable actions and transitions, an assumption that limits their ability to exploit modularity and local, context-dependent regularities present in many realistic environments. We propose a reinforcement learning framework using groupoids to capture local, state-dependent symmetries and support the dy- namic discovery of equivalence structures during interaction. The agent maintains orbit representatives together with transporters that map raw states to canonical forms, enabling learning and decision-making to be performed in a symmetry-reduced space while preserving local distinctions. Empirical results demonstrate that the proposed groupoid-based approach improves sample efficiency and convergence in dense and large-scale environments exhibiting strong partial symmetries, yielding substantial performance gains over standard Q-learning. These findings show that dynamically exploiting local symmetry provides a practical and mathematically principled route to scalable and generalisable reinforcement learning.