Modular system improves requirement quality checks using expert knowledge
MARQ: A Modular, LLM-Agnostic Requirement Quality Assessor Infused with Systems Engineering Knowledge
Artificial IntelligenceSoftware Engineering
Summary
Requirements guide engineers on what to build, but if they have mistakes, problems can spread during development. The authors show that large language models (LLMs) alone often miss many of these errors or are inconsistent. They created MARQ, a system that enhances any LLM by adding expert engineering knowledge focused on requirement quality. MARQ improves how well these models detect errors in requirements, though the exact improvements depend on the LLM used. This suggests combining expert knowledge with AI can make requirement checks more reliable, but human review remains important.
What this means in practice
- •For systems engineering teams: Use a modular tool that enhances any large language model to better detect defects in engineering requirements for complex products.
- •For software development teams: Integrate enhanced AI-driven requirement quality checking to catch specification errors early and reduce costly downstream fixes.
Authors
Jannatul Shefa, Alejandro Salado, Paul Wach, Taylan G. Topcu
Abstract
Requirements engineering (RE) is foundational to systems engineering (SE), as requirement defects can propagate downstream and adversely affect development outcomes. Although large language models (LLMs) are promising for requirement quality assessment, prior work shows that off-the-shelf models miss a large share of expert-identified defects, perform unevenly across quality criteria, and vary across executions. This study investigates whether infusing expert-derived SE knowledge can improve LLM-based requirement quality evaluation beyond off-the-shelf capability. We present MARQ, a modular, LLM-agnostic prototype that supplements LLMs with a criterion-specific knowledge stack operationalizing INCOSE requirement quality criteria and evaluated against off-the-shelf baselines on an expert-labeled NASA requirement set using Claude Opus 5, GPT-5.6 Sol, and the open-weight Gemma 3 12B, with 100 executions per model. MARQ achieves median F1 scores of 0.80 for Claude Opus 5, 0.62 for GPT-5.6 Sol, and 0.53 for Gemma 3 12B, compared with off-the-shelf baselines of 0.55, 0.52, and 0.18, respectively. However, the nature of these gains is model-dependent. Claude Opus 5 improves defect detection and false-alarm control; GPT-5.6 Sol gains detection and better execution-level consistency at the cost of more false alarms; and Gemma 3 12B reaches the performance of off-the-shelf proprietary models, although it remains highly variable. Criterion-level analysis shows gains concentrated in Correctness and Conformance, trade-offs between misses and false alarms within individual criteria, and model-dependent responses across several criteria. These findings position SE knowledge infusion as a promising model-agnostic pathway for strengthening trustworthy LLM-based RE support while underscoring the need for criterion-specific interventions and human-in-the-loop review.