Reranking large language model answers using paths in knowledge graphs
Reranking Answers of Large Language Models with Knowledge Graphs
Computation and LanguageArtificial Intelligence
Summary
Answering questions from big databases of facts is hard because there is too much information to quickly search through. This paper shows how to find smaller groups of related facts linked to possible answers by looking for the shortest connections between things the question talks about. The researchers created methods to pick the best answers from many possibilities using these smaller groups. They tested this on a big public database called Wikidata and also made a web tool that shows these connections visually and explains them in simple sentences.
What this means in practice
- •For chatbot developers: Improve answer quality by reranking chatbot responses using paths connecting entities in background knowledge graphs.
- •For knowledge base managers: Use automatically extracted subgraphs to better understand and refine connections relevant to user questions in large fact repositories.
Tested on one dataset.
Authors
Mikhail Salnikov, Hai Le, Olga Tsymboi, Ivan Lazichny, Dmitrii Iarosh, Egor Cheremiskin, Andrey Savchenko, Dmitry Simakov, Alexander Panchenko
Abstract
Answering natural language questions over knowledge graph data is challenging due to the vast number of facts, which can be difficult to process and navigate. One potential solution for this issue is to use mined subgraphs related to the query, although this process still requires extracting these subgraphs. This research presents a solution for extracting subgraphs related to entity candidates from a question-and-answer set, which can be obtained by inferring a large language model by calculating the shortest paths between entities. The proposed approaches detail various features that can be extracted from the subgraphs and reranking models to select the most probable answers from a list of candidates. Experiments were conducted on Wikidata to evaluate the effectiveness of the proposed approaches. This involved enumerating all the main feature types that can be extracted from mined subgraphs and a detailed analysis of the proposed features and reranking method combinations. In addition, a public web application that provides a useful web tool for studying the graph space between question and answer entities has been developed to work with subgraphs. This includes visualization of the extracted subgraph and automatic generation of natural language text to describe it.