Redistribution-based Cost Inference Improves Sparse Safe Offline RL
2026-08-12 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors address a problem in safe offline reinforcement learning, where usually detailed step-by-step costs are needed but in reality only a simple stop signal is given when something unsafe happens. They propose a method called Redistribution-based Cost Inference (RCI) that turns this sparse signal into detailed per-step costs so the learning algorithm can better understand what led to the unsafe event. They prove this method doesn't lose important information and helps train safer policies. Their tests on driving and robot tasks show their approach results in fewer safety violations compared to other methods, even with mixed data and noisy signals.
offline reinforcement learningtemporal credit assignmentreturn decompositioncost inferenceconstrained Markov decision processpolicy optimizationsafe reinforcement learningtrajectory-level feedbackcost criticlabel noise
Authors
Ebenezer Gelo, Geraud Nangue Tasse, Steven James, Benjamin Rosman
Abstract
Safe offline RL typically assumes access to dense per-step cost annotations, but in practice supervisors provide only trajectory-level stop-feedback: a binary signal at the first unsafe transition, with no per-step attribution. We frame this as a temporal credit assignment problem and propose the Redistribution-based Cost Inference (RCI) framework, which converts sparse stop-feedback into dense per-step costs via return decomposition, then trains a constrained offline policy on the augmented dataset. We show that return-equivalent redistribution preserves the feasible policy set and the optimal Lagrangian in a CMDP, establishing that the transformation is lossless in theory while yielding better-conditioned cost critic learning in practice. Experiments on highway driving and robotic manipulation demonstrate substantially lower violation rates than sparse and classifier-based baselines, with robustness to heterogeneous dataset compositions and label noise.