Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence
2026-08-31 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors study how large reasoning models (LRMs) can keep improving when human supervision is reduced, focusing on how rewards and learning experiences evolve. They describe a ladder from fully human-controlled learning to more independent, self-guided learning processes. The paper also discusses risks of more autonomous training, like models gaming their rewards or drifting off course, and suggests evaluating models by their abilities, the quality of feedback they get, and the experiences they learn from. Their work organizes current methods and challenges in building models that learn with less human help.
Large reasoning modelsReinforcement learningReward functionsHuman supervisionSelf-generated curriculaAutonomous learningReward hackingFeedback fidelityCurriculum learningExperience quality
Authors
Zhiqin Yang, Jingwen Fu, Yuhan Liu, Hengyu Liu, Yonggang Zhang, Kainan Cao, Zizhuo Zhang, Chenxin Li, Ruibin Yuan, Jiahao Pan, Jiankai Sun, Zhenyuan Zhang, Yibo Li, Yunlong Lin, Jing Xiong, Sida Lin, Bo Han, Wei Xue, Yike Guo
Abstract
Recent advances in large reasoning models (LRMs) have shown that reinforcement learning with verifiable rewards (RLVR) can substantially improve reasoning in mathematics and code, where outcomes can be checked automatically. Extending this progress to open-ended and agentic tasks remains difficult because reliable rewards are harder to obtain and direct human supervision cannot keep pace with the scale and complexity of model-generated experience. This paper studies how LRMs can continue to improve as human supervision gradually recedes from the learning loop. We examine two connected dimensions of this problem. The reward axis traces the development from per-instance human judgments to reusable verifiers and rewards that operate even without human feedback. The experience axis examines how learning can progress from human-curated tasks and environments toward self-generated curricula, constructed environments, and autonomous co-evolution. We connect these dimensions through a five-level ladder from L0 to L4 that identifies which parts of the learning process remain under continued human control. Our analysis further highlights the risks introduced by increasingly autonomous rewards and experience generation, including reward hacking, feedback drift, curriculum collapse, and environment errors. Consequently, we also provide the evaluation around three complementary objects: policy capability, feedback fidelity, and experience quality. This analysis provides a structured account of current approaches to scaling LRMs beyond human supervision and the open problems involved in developing self-sustaining learning systems toward superintelligence. Furthermore, we maintain a continuously updated \href{https://github.com/visitworld123/Awesome-Scaling-LRM-Beyond-Human-Supervision}{GitHub repository} to track the latest advances.