Dream-RSI improves AI exploration by learning from past searches
Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Computation and Language
Summary
Finding better solutions in complex problems is hard because AI agents struggle to explore efficiently. The authors introduce Dream-RSI, a method where the AI learns from its previous experiences by simulating past searches. This lets the AI improve its strategies without the usual slow and costly trial-and-error in the real world. Over time, this loop of learning and exploring helps the AI get better at discovery tasks in fields like math and computer engineering.
What this means in practice
- •For software engineers: Enhance automated algorithm tuning by iteratively improving exploration policies using previous search results to reduce trial-and-error costs.
- •For data center operators: Optimize GPU kernel performance by employing recursive exploration strategies that learn from past configuration outcomes to save evaluation time.
Authors
Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, Yunsong Guo
Abstract
Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce \textsc{Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, \textsc{Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.