CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents
2026-07-06 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address the problem that large language models (LLMs) struggle to remember very long conversations because they have limited context windows. They propose CompactionRL, a technique that helps LLMs summarize past interactions efficiently while still learning to complete tasks well. By training models with this approach, the authors show improved performance on coding tasks that need many steps. Their method helps existing open LLMs perform better by enabling them to work with compressed long histories of interactions.
large language modelscontext windowreinforcement learningcontext compactionlong-horizon taskssummary generationgeneralized advantage estimationPass@1agentic LLMscoding benchmarks
Authors
Yujiang Li, Zhenyu Hou, Yi Jing, Jie Tang, Yuxiao Dong
Abstract
Long-horizon agentic LLMs are increasingly limited by finite context windows, as extended interaction trajectories can exceed the maximum context length before a task is completed. Context compaction offers a natural solution by summarizing previous interaction states and continuing the rollout under a compressed context, but incorporating compaction into reinforcement learning remains underexplored. We propose CompactionRL, a reinforcement learning strategy to train long-horizon agentic LLMs with context compaction. Our approach jointly optimizes task execution and summary generation with token-level loss normalization and cross-trajectory generalized advantage estimation. This design enables the LLM agents to learn from compacted long-horizon trajectories. We train CompactionRL on top of open models and observe consistent performance gains on agentic coding tasks. CompactionRL enables the open GLM-4.5-Air model (106B-A30B) to achieve Pass@1 scores of 66.8% on SWE-bench Verified and 24.5% on Terminal-Bench 2.0, with absolute gains of 7.0 and 3.1 points, respectively. Built upon GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.8 points, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0, respectively. CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).