InfoPPO improves large language model reasoning by measuring information flow
Information-Time Proximal Policy Optimization
Machine LearningArtificial Intelligence
Summary
Large language models learn to generate text step-by-step, but not all steps carry the same amount of useful information. This paper introduces InfoPPO, a new method that measures progress based on how much information is gained rather than just counting words. By doing this, their method helps models learn better over long sequences and improves their accuracy on complex math problems. The authors tested InfoPPO on several benchmarks and showed it works better than older methods.
What this means in practice
- •For ai engineers: Improve long-form reasoning performance of language model based AI systems by using information-density-aware policy updates.
- •For narrative generation teams: Develop more coherent extended text outputs by adjusting learning progress based on information flow rather than token count.
Authors
Yongcheng Zeng, Xinyu Cui, Yan Song, Guoqing Liu, Hongsheng Xin, Kaike Zhang, Cheng Deng, Kun Zhan, Jian Ying, Jian Zhao, Haifeng Zhang, Jun Wang
Abstract
RLVR has substantially improved the reasoning capabilities of LLMs. However, existing methods typically parameterize temporal progression in the Markov Decision Process by token-by-token generation, despite the highly non-uniform information flow along autoregressive trajectories. In this paper, we propose InfoPPO, which reparameterizes temporal progression using information density rather than raw token count. This reparameterization induces a common state-dependent structure for both temporal credit propagation and policy updates. InfoPPO restores the effectiveness of non-trivial discounting in long-horizon reasoning, retaining effective-horizon contraction while avoiding excessive attenuation of terminal supervision over long token sequences. Moreover, the information-time policy-improvement analysis naturally leads to a state-dependent update constraint, which we implement through adaptive clipping. By adapting the clipping threshold at each token position to the information density of its corresponding state, this mechanism enables more targeted policy updates while preserving proximal control. Theoretically, we extend performance-difference and policy-improvement analyses to the information-time MDP, deriving a policy-improvement lower bound when policy changes are regulated by information density. We further connect the general information-time analysis to practical LLM policy optimization by relating state-wise information density to local policy movement, while also providing theoretical grounding for the adaptive update mechanism. Experiments on Qwen3 models demonstrate consistent gains over competitive baselines across five challenging competition-style mathematical reasoning benchmarks. InfoPPO also maintains stable accuracy and response length across non-trivial discount settings under which token-time PPO deteriorates.