Reinforcement learning algorithms improve themselves with ai guidance

RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms

Artificial Intelligence

Summary

Finding better ways for AI agents to learn is hard because changing many parts at once can confuse the system. The authors created RLDiscover, a method where AI slowly tweaks its own learning steps and checks if those changes help by trying them out carefully. Their tests show big improvements in popular AI learning methods, using much less computing power than usual. The new methods even work well on new tasks, highlighting potential for AI to improve its own learning processes over time.

What this means in practice

  • For machine learning engineers: Improve training efficiency and performance of reinforcement learning agents by using self-evolving algorithm components tailored to specific tasks.
  • For autonomous robotics teams: Enhance robot control algorithms to adapt through AI-driven self-optimization of learning strategies for diverse environments.

Authors

Haoran Li, Zengle Ge, Xiaomin Yuan, Yui Lo, Songlin Zhou, Jiahua Ying, Haoxin Li, Qianhui Liu, Yuanhang Liu, Jiaqun Liu, Guokai Chen, Mingju Chen, Ruinan Wang, Annan Li, Jianmin Wu, Dawei Yin, Dou Shen

Abstract

LLM-guided program evolution has enabled discoveries in mathematics and computational optimization, raising the prospect of reinforcement learning (RL) algorithms that self-evolve to improve how agents learn. However, realizing this prospect faces two obstacles. Joint search over coupled algorithmic components is difficult to scale: simultaneous changes can disrupt learning, while isolated changes overlook their dependencies. Evaluating candidate algorithms also requires costly training, with fitness remaining uncertain across random seeds. We introduce RLDiscover, a framework for the self-evolution of model-free deep RL algorithms. Progressive Co-Evolution advances from targeted component edits to joint evolution, while Progressive Probabilistic Evaluation balances search breadth and evaluation fidelity through staged training and repeated evaluation. Experiments across SAC, PPO, and DQN on four benchmark suites show substantial improvements in mean return, with per-family median gains of 32%-84% and a peak return ratio of approximately 363x over a near-zero baseline. These gains include transitions from failed learning to successful task completion, and improvements persist when evolution starts from stronger open-source implementations. On measured SAC locomotion runs, evaluation uses approximately one-fifteenth the estimated compute required to fully evaluate the same candidate pool. Remarkably, independent searches repeatedly discover interpretable combinations of adaptive robust losses, progress-dependent value targets, and running statistics, with selected programs transferring to unseen tasks. These findings point toward a broader role for self-evolution in AI: discovering interpretable algorithms that improve how agents learn.