Semantic parsing workflow roles shape translation success and errors
Before Blaming the LLM, Diagnose the Workflow: A Role-Level Difficulty Profile for Translating Natural Language into Ontology-Grounded Statements
Artificial Intelligence
Summary
Translating everyday language into precise computer code that fits specific domains can be tricky. The authors show that problems often come from various parts of the process—not just the computer model itself. They identify key roles in the workflow, such as input quality, how much background context is available, and how well the output is checked for errors. By examining these roles with a special profile, people can better figure out where issues occur and improve the whole translation process.
What this means in practice
- •For software developers: Improve debugging of natural-language-to-code translation by analyzing input, generation, and validation workflow roles separately.
- •For workflow engineers: Guide workflow redesign to enhance domain language translation quality using a role-level difficulty profile.
A position paper. It proposes an approach and reports no results.
Authors
Liang Zhang, Benoît Daireaux, Sergey Alyaev
Abstract
Natural-language-to-domain-specific-language (NL-to-DSL) semantic parsing systems increasingly translate informal user descriptions into structured artifacts such as Structured Query Language (SQL) queries, Application Programming Interface (API) calls, workflows, formal specifications, and ontology-grounded domain-specific language (DSL) statements. This paper focuses on ontology-grounded Assertion Box (ABox) DSL generation: the translation of natural language domain descriptions into assertion-level DSL statements grounded in a pre-existing ontology vocabulary. In this setting, syntactic validity is often easier to achieve than semantic adequacy. Failures are frequently attributed to weak generator models, but recurring sources of difficulty are also structural: insufficient input, weak domain grounding, output template and vocabulary gaps, weak validation, and non-local repair. We propose a role-level difficulty profile for diagnosing difficulty in ontology-grounded NL-to-DSL semantic parsing at the workflow level. The profile organizes diagnostic variables across five recurring workflow roles: Input, Generator, Output, Validator, and Repairer. The seven variables are input sufficiency, context availability, grounding complexity, template fit, vocabulary coverage, validation strength, and error observability. Practitioners construct the difficulty profile through qualitative judgment over available evidence. This experience-informed, non-exhaustive difficulty profile is intended to support workflow diagnosis and guide redesign or human-in-the-loop intervention.