GPT 4 chatbot helps people explore software project data easily
A Tool-Augmented, GPT-4 Chatbot for Real-Time Repository Data Analysis
Artificial IntelligenceSoftware Engineering
Summary
Lots of information about software projects is stored in online repositories, but it can be hard for people who are not experts to find answers from all that data. The authors created a chatbot that uses GPT-4 to understand questions about the project, pick the right tool to get the needed data, and then give detailed answers. This chatbot works differently by first figuring out exactly what the user wants before asking GPT-4, rather than searching through documents first. Their tests showed this way helps make complex project information clearer and more useful for both technical and non-technical users.
software repositoryGPT-4chatbotprompt engineeringtool selectioncode contributionsbug reportspull requestsissue trackingdata analysis
Authors
Muhammad Jawad Chowdhury, Md. Sakib Khan
Abstract
Software repositories contain vast amounts of data on code contributions, bug reports, and project activities, yet this information remains challenging for non-technical stakeholders and developers to access due to limited expertise in querying repositories. To address this, we introduce a novel chatbot architecture leveraging OpenAI's GPT-4 model for automated extraction and analysis of repository data. In contrast, our architecture takes a structured path first by parsing the user's query to extract relevant parameters, then selecting the correct tool to employ based on that analysis, and finally invoking the GPT-4 model to create a highly detailed response. In contrast to previous work based on multi-component systems with embedding models and document retrievers, our architecture inverts the process by relying on prompt engineering and tool selection to fit with the query intent. To validate our approach, we conducted experiments on various question types, including Issues, Pull Requests, Commits, Compound Questions, and General Repository Information, evaluating our target prompts' ability to improve the accuracy of responses from the model. Beyond demonstrating the utility of this architecture to a diverse set of users, our findings suggest that this architecture can make repository data more accessible to technical and non-technical audiences through the production of actionable insights.