Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback
2026-06-29 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors created NL-PDDL-Bench, a test set to help computers turn natural language instructions into formal plans that can be checked and executed safely. They also designed a system that lets the computer fix its own plan instructions when there are errors, using feedback from a planner. Their method combines some smart training techniques and on-the-fly corrections without needing constant planning during learning. Testing showed their approach makes planning more accurate and reliable across different tasks and harder problems. This work aims to make large language models safer and more dependable when used for automated decision-making.
Natural Language ProcessingPDDLAutomated PlanningLarge Language ModelsLow-Rank AdaptationDirect Preference OptimizationPlan VerificationBenchmarkingPlanner-in-the-loopFine-tuning
Authors
Jiamei Jiang, Jiajing Zhang, Feifei Mo, Linjing Li, Daniel Zeng
Abstract
Planning often requires symbolic specifications that are both executable and verifiable. For large language models deployed in autonomous or decision-support systems, failures in such formalization may lead to unverifiable decisions, execution failures, or unsafe downstream behavior. We present NL-PDDL-Bench, a multi-domain benchmark for natural-language-to-PDDL specification construction with planner-verified executability and controlled difficulty scaling by object count. We further propose a planner-in-the-loop framework that uses validator and planner diagnostics to revise non-executable specifications through localized edits. Building on this infrastructure, we develop a planner-grounded optimization recipe that combines parameter-efficient Low-Rank Adaptation supervised fine-tuning, offline planner-derived preference pairs for Direct Preference Optimization, and inference-time planner-in-the-loop repair, without requiring online planner calls during training. We also provide a unified evaluation suite for parseability, solvability, specification similarity, and outcome-aware plan-level consistency against planner references. Experiments on representative model families show substantial gains in planner success and plan-level agreement, with improved robustness under difficulty scaling and cross-domain variation. These results highlight the value of externally verifiable formalization for reliable deployment of LLMs in safety- or security-sensitive planning systems. Code and data are available at: https://github.com/ibasicplan/NL-PDDL-Bench