On the Prospects of Dynamic LLM Conversations in Software Development
2026-08-31 • Software Engineering
Software EngineeringArtificial Intelligence
AI summaryⓘ
The authors studied how small changes in the way chat-based large language models (LLMs) interact with developers affect their experience. They tested three groups of computer science students using LLMs for web development: one received context-aware help, another got proactive suggestions, and the last had no extra help. Their results showed no harmful effects from the interventions and a slight increase in satisfaction with proactive support. This suggests that even small improvements in LLM guidance can positively impact developers.
large language modelsLLMchat-based interactiondeveloper experienceintent-based augmentationproactive suggestionssoftware developmentuser studyinteraction logslongitudinal study
Authors
Annemarie Wittig, Alina Mailach, Janet Siegmund, Norbert Siegmund
Abstract
Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a \emph{context}-aware group received intent-based conversation augmentation, (2) a \emph{proactive} group received follow-up suggestions and tailored advice, and (3) a \emph{control} group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the \emph{proactive} treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.