Learning robot constraints safely without risky trial and error
Learning Unknown Constraints without Unsafe Data via Optimality and Counterfactual Regularization
RoboticsMachine Learning
Summary
Robots often need to learn rules or limits from expert examples to operate safely. The authors present a way to learn these unknown rules using only examples that follow them, without needing the robot to try unsafe actions. They do this by modeling the robot’s movements and imagining better but unsafe actions nearby to figure out what rules make those unsafe. Their method combines safety and flexibility better than previous approaches and works well on complex robot tasks.
What this means in practice
- •For robot developers: Train robots to identify safety limits from expert behavior without unsafe exploratory actions.
- •For autonomous vehicle engineers: Improve constraint learning in vehicles using offline expert data, reducing the need for risky on-road testing.
Authors
Zhouyu Zhang, Chih-Yuan Chiu, Glen Chou
Abstract
Learning from demonstrations (LfD) provides a framework for inferring unknown constraints from locally optimal, constraint-satisfying expert behavior. Existing approaches largely fall into two paradigms, constrained inverse optimal control (CIOC) and inverse constrained reinforcement learning (ICRL). CIOC exploits optimality conditions such as the Karush--Kuhn--Tucker (KKT) conditions but typically assumes known dynamics and structured constraint representations. Meanwhile, ICRL accommodates complex unknown constraints and unknown transition dynamics but often requires extensive online exploration, during which unsafe constraint violations may occur. In this work, we introduce Counterfactual KKT (CF-KKT), a constraint learning framework that leverages learned dynamics and locally optimal demonstrations to recover unknown constraints without requiring known dynamics or additional risky exploration, thereby combining the data efficiency and safety advantages of CIOC with the flexibility of ICRL. First, we use a locally learned differentiable dynamics model to impose KKT-inspired optimality conditions directly on the demonstrations. Second, we use the learned dynamics to generate reward-improving counterfactual behaviors near the demonstrations, revealing behaviors that would be preferable in the absence of the unknown constraint and thus providing synthetic infeasible data. When the constraint parameterization is known, the same learned-dynamics framework enables direct CIOC-based parameter recovery, and we characterize its sensitivity to dynamics misspecification. Across high-dimensional robotic control tasks, our approach learns neural constraint representations with improved safety and data efficiency relative to state-of-the-art offline ICRL baselines.