Policies naturally absorb trainable action offsets they can express
Action Shaping: Policies Absorb What They Can Express
Artificial IntelligenceMachine Learning
Summary
This paper looks at how machine learning policies handle small changes added to their actions during training. The authors show that if a policy can exactly recreate these changes in its own output, it will absorb them and leave its decisions unchanged when those changes are removed later. They provide a clear condition for when this happens and how to detect it. This helps understand a common practice where adjustments are added during training and then removed during use, something without a formal guarantee until now.
What this means in practice
- •For reinforcement learning engineers: Ensure added action offsets during training can be safely removed at deployment if the policy can reproduce them exactly.
- •For robotics control teams: Use trainable action offsets to stabilize learning while guaranteeing final robot behaviors remain unaffected after offset removal.
Authors
Yanjun Chen, Jinghan Wang, Xiaoyu Shen, Wenjie Li, Wei Zhang
Abstract
Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.