Reinforcement learning boosts large language models self learning skills
Incentivizing Self-Learning in Large Language Models via Reinforcement Learning
Computation and Language
Summary
Large language models often rely on pre-collected facts or external lookups to answer hard questions. This paper explores how these models can teach themselves by reading and understanding textbooks without direct help. The authors created a setup where a model uses reinforcement learning to reward itself for figuring out math problems by studying textbook sections. Their approach improved the model’s ability to solve problems in familiar and new subjects, showing that teaching models to learn independently from long, detailed materials can make them better problem solvers.
What this means in practice
- •For software developers: Improve AI assistants by enabling them to learn new domain knowledge autonomously from large textbooks instead of relying on fixed datasets.
- •For automated tutoring systems: Enhance educational software to independently study and generate solutions for complex problems without predefined answers, adapting to diverse subjects.
Authors
Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen, Wenwei Zhang
Abstract
Large language models (LLMs) draw on parametric knowledge and retrieval-augmented generation (RAG) to solve knowledge-intensive tasks. Yet problem solving in unfamiliar domains calls for tightly coupled knowledge acquisition and reasoning, beyond factual retrieval and aggregation or learning solely from a preassembled context. We study self-learning -- autonomously retrieving, studying, and applying knowledge from long-form reference materials on the fly. To investigate this capability, we formulate a textbook-based mathematical problem-solving task. LLMs must navigate textbook exposition, learn and apply relevant principles, and construct proofs for exercises, with reference solutions withheld. We introduce a BookNavigator interface to support this human-inspired study process and incentivize self-learning through reinforcement learning (RL) with fine-grained rewards. Experiments show that our method improves performance across in-domain and out-of-domain textbooks and subjects, raising Intern-S2-Preview-35B's overall proof accuracy from 51.5% to 79.5% and outperforming RL-trained textbook-free and fixed-retrieval baselines. These gains extend to complex long-context benchmarks involving different task types, with improvements over the base model of 1.7 and 2.3 percentage points on BrowseComp-Plus and CL-bench, respectively. Together, these findings suggest that RL can foster a domain-agnostic self-learning strategy, highlighting the potential of self-learning from long-form corpora to strengthen broader problem-solving capabilities on complex, long-context tasks.