Reinforcement learning tracks hidden changes with automata inference
Temporal-Causal Inference for Reinforcement Learning via Automata Learning
Machine Learning
Summary
Some environments change suddenly in ways that aren’t directly visible but affect how to act best. The researchers study how to teach a computer agent to learn both what to do and when these hidden changes happen, even though the agent can't see them directly. They use a method that builds a simple machine (a deterministic finite automaton) to guess the hidden pattern that causes the change and improve actions over time. Their approach learns the true hidden pattern reliably and performs as well as if it knew the hidden information all along. They tested this idea on examples like directing traffic signals and medical therapy in a grid world.
What this means in practice
- •For traffic management engineers: Design traffic signal controllers that adapt to unobservable timing changes in traffic flow using inferred temporal patterns to improve traffic efficiency.
- •For biomedical simulation developers: Simulate and optimize treatment policies in genetic therapies where disease states change unpredictably and are not directly observable.
Authors
Jan Corazza, Daniil Kaminskyi, Simon Lutz, Patrick Nossol, Hadi Partovi Aria, Zhe Xu, Daniel Neider
Abstract
We consider reinforcement learning in environments with dynamics that undergo an irreversible phase transition governed by a hidden temporal pattern. The agent observes the base state but cannot observe the phase directly. We formalize this problem as a two-phase non-Markovian decision process and introduce Temporal-Causal Inference for Reinforcement Learning (TCIRL), a framework that jointly learns a control policy and infers the hidden temporal cause of the phase transition. TCIRL maintains a hypothesis deterministic finite automaton (DFA) to track what phase is active and refines it via counterexample-driven SAT-based synthesis. We prove that the hypothesis converges almost surely to a DFA recognizing the true cause language on all attainable label sequences, yielding an optimal policy for the original non-Markovian decision process. Experiments on a genetic therapy gridworld and a traffic signal environment show that TCIRL recovers the correct cause DFA and matches the full-information baseline in both domains.