CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs
2026-07-06 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors identify that large language models often make plans that don’t follow the rules or constraints of a task, which makes them less reliable. To fix this, they created a new training method called Constraint-Aware Reinforcement Learning (CARL) that teaches these models to pay more attention to constraints by giving rewards when they follow the rules and penalties when they don’t. Their approach works without needing extra tools and can be used with different reinforcement learning methods. Tests show that CARL helps models stick to constraints much better than previous methods.
Large Language ModelsConstraint SatisfactionReinforcement LearningReward FunctionTask PlanningFine-TuningConstraint-Aware LearningModel ReliabilityBlocksWorldTravelPlanner
Authors
Qiuyi Qi, Jinjian Zhang, Mutian Bao, Tian Liang, Guocong Li, Dongnan Liu, Wei Zhou, Jie Liu, Ming Kong, Linjian Mo, Feng Zhang, Qiang Zhu
Abstract
Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process. While existing approaches attempt to mitigate this by relying on external tools or task decomposition, they fail to enhance the model's intrinsic constraint awareness. To address this, we propose Constraint-Aware Reinforcement Learning (CARL), a novel RL framework designed to strengthen LLMs' intrinsic focus on constraints. CARL introduces a constraint-aware reward by comparing the model's output distributions under constrained and unconstrained inputs, encouraging constraint focus and penalizing neglect. Compatible with various RL frameworks and requiring no external solvers or top models, CARL enables scalable, end-to-end constraint-aware planning. Extensive experiments on BlocksWorld, TravelPlanner, and T-Eval demonstrate that CARL significantly outperforms standard Reinforcement Fine-Tuning (RFT) baselines and state-of-the-art reasoning models, exhibiting a markedly increased focus on constraints.