Safe-RULE: Safe Reinforcement UnLEarning
2026-06-08 • Machine Learning
Machine LearningArtificial IntelligenceCryptography and SecurityRobotics
AI summaryⓘ
The authors study a way to keep AI systems safe when learning from past data without interacting with the environment again. They point out that if bad or harmful data is mixed into the learning dataset, the AI can learn unsafe behaviors. To fix this, they introduce a method called Safe-RULE, which helps the AI 'unlearn' the bad data without starting over or needing the original environment. Their tests show this method improves safety even when the data was attacked.
Offline Reinforcement LearningSafe Reinforcement LearningData Poisoning AttackUnlearningPolicy LearningSafety ConstraintsRoboticsBenchmark Tasks
Authors
Shixiong Jiang, Taozheng Zhu, Fanxin Kong
Abstract
Offline safe reinforcement learning (Safe RL) enables policy learning without online interactions, making it suitable for safety-critical systems such as robotics systems. However, its reliance on static datasets exposes offline Safe RL to data poisoning attacks, where adversaries inject malicious samples that compromise safety and induce unsafe policy behavior. In this work, we propose a new learning paradigm, named safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data without retraining from scratch or requiring access to the original training environment. We further extend reinforcement unlearning to offline Safe RL by explicitly accounting for both task performance and safety constraints during the unlearning process. Experiments across benchmark Safe RL tasks demonstrate that our approach effectively enhances safety performance against data poisoning attacks.