Context operations improve systems design modelling with large language models

Context operations to architecture modelling output from large language models and evaluation criteria for their use in systems engineering design

Artificial IntelligenceSoftware Engineering

Summary

Designing complex systems is often slow and complicated. This paper offers a way to structure how large language models (LLMs) are used in engineering design by organizing different types of information like policies and previous references alongside user questions. The authors also propose a method to check if the model’s outputs really match what the designers want. This helps engineers trust and make better use of AI tools when planning system architectures.

What this means in practice

  • For systems engineers: Structure and evaluate AI-generated system architecture designs by integrating contextual prompts and checking if model outputs align with design goals.
  • For software architects: Use formal context assembly operations to improve the quality and traceability of architecture models produced by AI-assisted design tools.

Authors

Vinicius Kaster Marini, Petter Krus

Abstract

The development of generative artificial intelligence resources enables opportunities of speeding up systems and engineering design work. This contribution introduces a framework of formal operations for assembling context in LLM-based engineering design. This framework involves the assembly of modular context units, including policy prompts, reference units with persistence, and user questions with prompt vectoring. This approach enables the systematic structuring of interactions with generative models. A formal method for evaluating modelling-as-code LLM outputs is also presented, which enables the evaluation of compliance to intent from LLM answers and thereby asses the support from LLMs for systems architecture modelling.
View PDFOpen arXiv