How Do You Choose Your AI Component? An Interview Study of Secure AI Integration in Practice
Software EngineeringArtificial IntelligenceCryptography and SecurityInformation RetrievalMachine Learning
Summary
The authors studied how software developers choose and use AI models in their projects and found that they mostly focus on things like accuracy, cost, and features rather than security. They noticed that lessons from traditional software security, especially in managing software parts safely, are often ignored when it comes to AI components. This can make applications open to risks like bad actors or leaking sensitive data. The authors suggest that security should be considered from the start and throughout building software that uses AI.
Authors
Mahzabin Tamanna, Elizabeth Lin, Sparsha Gowda, Laurie Williams, Dominik Wermke
Abstract
The increasing adoption of Large Language Models (LLMs) as AI components in modern software systems introduces distinct security risks to the software supply chain. While many considerations and safety mechanisms are in place for components of the traditional software supply chain, the recent rapid adoption of AI components and platforms has overlooked these hard learned lessons. Selecting and integrating AI models without clear guidance on how these choices affect system security may leave applications vulnerable to threats, such as malicious components, data leakage, and unintended behavior. The goal of this study is to understand practitioners' decision making process and security considerations in selecting and integrating AI components through an exploratory semi-structured interview study. Toward this goal, we conducted semistructured interviews with 22 software developers, architects, and AI practitioners across diverse organizations about how they integrate AI components into their software. Our analysis finds that practitioners' model selection is predominantly driven by functional criteria, including performance, accuracy, cost, and specific features, e.g., tool calling or multimodal support, while security is rarely considered as an evaluation criterion. We observe a consistent lack of security concern throughout the AI component integration process, with established software supply chain lessons overlooked or ignored. The industry is repeating the historically costly mistakes of early software dependency management, prioritizing rapid reuse and availability over security and provenance. We distill our findings into actionable recommendations for AI adopters, model providers, and researchers, advocating for a proactive, security-by-design approach that integrates security evaluation into component selection and sustains it throughout the software development lifecycle.