Large language models accurately translate game tester goals to planning language
Which LLM is Best for Translating Natural Language Goals to PDDL
Artificial Intelligence
Summary
It's hard for non-experts to tell computers exactly what to do using formal languages. This paper looks at how well big AI models can convert simple English requests from video game testers into a special computer language called PDDL that computers use to plan actions. The authors tested six top models and found most did really well, with one called Gemini 2.5 Flash being the most accurate. However, sometimes the AI struggled with ambiguous language or specific details. This shows AI is getting better at understanding human goals but still faces some challenges.
What this means in practice
- •For video game developers: Convert informal testing goals into formal planning targets automatically for better test automation.
- •For automated planning engineers: Use AI to translate natural language objectives into formal planning languages without expert intervention.
Authors
Tomas Balyo, Lukas Chrpa, G. Michael Youngblood
Abstract
Bridging the gap between human intent and machine execution remains a challenge in automated planning, where expressing goals in formal languages like PDDL restricts accessibility to non-experts. This paper empirically evaluates whether current Large Language Models (LLMs) can reliably translate natural language testing goals, written in informal language by video game testers, into well-formed PDDL targets suitable for classical planning. We present a carefully designed prompt template, integrating insights from iterative experimentation, aimed at maximizing both accuracy and response coherence from multiple state-of-the-art LLMs. Six contemporary models are systematically assessed on correctness, speed, and error tendencies using real-world, domain-specific benchmarks. All models demonstrate high correctness, exceeding 92\%, with Gemini 2.5 Flash achieving the highest accuracy at 96\% and the lowest incidence of false positives, while GPT-4.1 leads in response speed. Despite these advances, critical distinctions exist in model performance, and occasional failures arise from language ambiguity and limitations in domain representation. Our analysis underscores both the significant progress and ongoing gaps in enabling LLMs to act as robust bridges between natural language objectives and automated planning pipelines.