Developers use few settings for large language model inference
How Developers Configure LLM Inference: A Large-Scale Study of Real-World Practices
Software Engineering
Summary
Large language models (LLMs) need many settings when they are used in software, but it is not clear which ones developers actually change. The authors studied thousands of open-source projects and found that most developers only pick a few options when configuring LLMs. Although most available options are used somewhere, most are rare and only a small number are commonly chosen. The way settings are combined varies a lot, and different applications use different settings patterns.
What this means in practice
- •For software engineers: Focus backend configuration efforts on frequently used LLM inference options and typical setting combinations found in real projects.
- •For platform developers: Improve LLM backend documentation and configuration tools by prioritizing commonly adopted options and patterns identified from large open-source usage.
Authors
Nada Zine, Matias Martinez, Romain Rouvoy, Clément Quinton
Abstract
LLMs are increasingly integrated into software systems, making deployment and inference configuration a recurring software engineering task. Inference backends expose many options that affect model loading, generation, serving, and request processing. Yet little is known about which of these options developers explicitly configure in practice. In this paper, we present a large-scale empirical study of LLM inference configuration practices in open-source software using Hugging Face Transformers, vLLM, and the OpenAI API. We analyze 5,398 GitHub repositories using at least one of these backends and identify 345,117 explicitly configured option occurrences. Our results show that developers use only a small part of the available configuration space regularly. Although 98.8% of the documented backend-specific options appear at least once, 78.9% are configured in at most 10% of repositories using the corresponding backend. Configurations are also small, with a median of 2 options for Transformers and 4 for vLLM and OpenAI, but the combinations of options vary widely: 56.8-61.6% of multi-option signatures appear in only one repository. We also observe recurring option values as well as substantial variation in how values are chosen. Finally, configuration practices differ across application contexts, with option-adoption differences reaching 30.7 percentage points. These findings characterize the configuration surface developers make explicit rather than the importance or optimality of individual options. They provide empirical evidence for prioritizing future studies of inference configuration and for improving backend documentation and configuration support.