Reinforcement learning improves robot insertion tasks with potential fields
Potential-Field Action Representation for Reinforcement Learning in Contact-Rich Manipulation
RoboticsArtificial Intelligence
Summary
Robots that handle objects by touching and inserting parts often need to learn complex movements, which can be difficult to teach. The authors introduce a new way for robots to learn these movements by using something called artificial potential fields, which guide the robot’s actions indirectly rather than controlling every small move. Their approach helped the robot complete a tricky peg-in-hole task perfectly in simulations and worked well on a real robot without extra training. This method also made the robot’s movements smoother and less forceful compared to other techniques.
What this means in practice
- •For industrial automation engineers: Control robots performing delicate insertion tasks more reliably by using potential-field based action control to improve success rates and smoothness.
- •For robotics system integrators: Deploy learned manipulation policies trained in simulation directly on real robots without needing fine-tuning, reducing setup time for new tasks.
Authors
Xinyu Liu, Gökhan Solak, Arash Ajoudani
Abstract
Model-free reinforcement learning can acquire contact-rich robotic manipulation skills through trial-and-error interaction, but it often requires the policy to learn both task strategy and low-level motion generation. In this setting, the action representation is critical because it determines how policy outputs are converted into robot motion, shaping both exploration and physical execution. Direct Cartesian command interfaces require the policy to generate motion at every decision step, coupling task-level adaptation with continuous low-level control and increasing the learning burden. We propose PA-RL, a reinforcement-learning framework that uses artificial potential fields as the action representation. Instead of commanding motion directly, the policy adapts the parameters of an energy-like potential field, which generates a state-dependent guidance direction executed through a Cartesian impedance controller. We evaluate PA-RL on peg-in-hole insertion, a representative contact-rich task with nonlinear dynamics and discontinuous contact transitions. In simulation, PA-RL is compared with Cartesian velocity, Cartesian pose, and variable-impedance action spaces using the same RL algorithm. PA-RL is the only method to reach a 100% evaluation success rate within the allotted training time, while the best baseline reaches 92.6%. It also reduces joint-torque variation by 55.4% and Cartesian acceleration variation by 70.8% relative to the best baseline, without explicit motion-quality penalties in the reward. The simulation-trained policy further completes 9/9 real-robot insertions without fine-tuning, demonstrating the deployment feasibility of the learned potential-field interface.