Evaluation and repair improve language model planning problem generation
Grounded Evaluation and Repair for NL-to-PDDL Problem Generation
Artificial Intelligence
Summary
Translating description text into formal planning problems is tricky because tools often only check if the problem is syntactically correct or solvable, missing if it truly matches the original description. The authors explored a process that uses large language models to generate problems, then checks and repairs them using several steps, including parsing, planning, domain rules, and feedback loops. Their findings show that a problem can be solvable but still not faithful to the described task, and guiding repairs improves accuracy though challenges remain. They also showed that fully matching known problem sets is still hard even with better success rates.
What this means in practice
- •For automated planning engineers: Create more accurate planning problem instances from natural language descriptions by integrating generation, validation, and repair processes.
- •For software developers building ai assistants: Improve natural language understanding pipelines by adding iterative feedback and repair steps based on domain rules and problem validation.
Authors
Joana Rosa, Pedro Santos, Valdemar Oliveira, Romão Silva, L. Miguel Silveira, Bruno Martins
Abstract
Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standard evaluation criteria such as syntactic validity or planner success can substantially overestimate faithfulness to the described task: a generated problem may be parseable and solvable while misrepresenting the intended initial state, goal, object structure, or optimization target. This paper studies an end-to-end NL-to-PDDL pipeline that combines LLM generation, checks in terms of PDDL parsing, planning and validation, a domain-conformance checker, an LLM critic, and iterative repair. Fine-grained repair feedback is constructed from the domain description, the generated problem, the natural language problem description, and operational diagnostics. Reference-based comparisons against curated benchmark PDDL problem descriptions are used for post-hoc benchmark analysis, and these offline checks include renaming-invariant structural matching and semantic equivalence, where domain support is available. Across Planetarium, AutoPlanBench, and curated PDDL~2.1 problems, results show that operational success and benchmark-reference reconstruction can diverge substantially. Results also show that structured repair can be useful, and that PDDL~2.1 remains challenging for reference reconstruction, even when operational success improves.