Audit layer reveals why reinforcement learning meets logic goals
Satisfaction Is Not Explanation: Auditing Vacuity and Training Influence in Temporal-Logic-Guided Reinforcement Learning
Software Engineering
Summary
When a computer learns to follow rules expressed in logic, it may appear to meet them but actually just get lucky or take shortcuts. The authors found that just knowing the computer passed the rules is not enough to know why it did so. They created a method that checks if the parts of the rules truly mattered during learning or were just coincidentally satisfied. This helps reveal if the computer really learned what was intended or just avoided hard-to-check parts.
What this means in practice
- •For autonomous system developers: Detect if reinforcement learning policies genuinely use specific logical rules or just bypass them to improve task success.
- •For robotics engineers: Evaluate and debug robot behavior to ensure compliance with formal temporal-logic specifications beyond just passing tests.
Authors
Lorenzo Bacchiani
Abstract
A reinforcement learning policy that satisfies its temporal-logic specification has passed a test, not an assurance argument. The clause that matters to a reviewer may never have mattered to training: it may have been avoided entirely, forced by the environment regardless of what the policy learned, or redundant next to the ordinary task reward. Satisfaction probability and task return cannot tell any of this apart. This paper introduces an audit layer that can. It measures whether a specification clause was actually exercised, whether that role was forced or chosen, and whether the obvious way to test causation, weakening the clause and retraining, is even valid. Often it is not: we prove that comparable weaker/stronger training objectives can share perfect optima under standard acceptance-derived rewards, show related ablation hazards across a large corpus of published specifications, and then show that a properly designed intervention detects the effect it should. Across standard reinforcement learning benchmarks and published external artifacts, the audit layer separates six regimes that a single satisfaction number collapses into one. A policy that satisfies its specification has answered whether. This paper asks why.