Chatbot using GPT-4 helps everyone access software repository data
A Tool-Augmented, GPT-4 Chatbot for Real-Time Repository Data Analysis
Artificial IntelligenceSoftware Engineering
Summary
Software project data like bug reports and code changes are hard for many people to explore without technical know-how. The authors designed a chatbot that breaks down user questions to pick the right tools before using GPT-4 to give detailed answers. This approach helps people with different backgrounds get useful insights from complex repository data. Their tests showed it works well with various question types and can make software project information easier to understand.
What this means in practice
- •For software development teams: Enable developers to quickly analyze and extract insights from project repositories without detailed query expertise.
- •For project managers: Allow non-technical managers to access and understand real-time data from code repositories through conversational queries.
- •For software analytics vendors: Offer chatbot-driven repository data analysis as a service to help clients interact naturally with their development data.$Commercial implications: Enables creation of marketable AI chatbots that provide actionable insights to customers from their own project data.
Authors
Muhammad Jawad Chowdhury, Md. Sakib Khan
Abstract
Software repositories contain vast amounts of data on code contributions, bug reports, and project activities, yet this information remains challenging for non-technical stakeholders and developers to access due to limited expertise in querying repositories. To address this, we introduce a novel chatbot architecture leveraging OpenAI's GPT-4 model for automated extraction and analysis of repository data. In contrast, our architecture takes a structured path first by parsing the user's query to extract relevant parameters, then selecting the correct tool to employ based on that analysis, and finally invoking the GPT-4 model to create a highly detailed response. In contrast to previous work based on multi-component systems with embedding models and document retrievers, our architecture inverts the process by relying on prompt engineering and tool selection to fit with the query intent. To validate our approach, we conducted experiments on various question types, including Issues, Pull Requests, Commits, Compound Questions, and General Repository Information, evaluating our target prompts' ability to improve the accuracy of responses from the model. Beyond demonstrating the utility of this architecture to a diverse set of users, our findings suggest that this architecture can make repository data more accessible to technical and non-technical audiences through the production of actionable insights.