KGCaRe: Explainable Complex Conditional Question Answering using Automatic Knowledge Graph Construction and Context Retrieval with LLMs
2026-08-10 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors explore improving how AI models answer complicated 'if-then' questions, especially in specialized areas where usual methods struggle. They developed KGCaRe, which combines finding information using vectors with detailed reasoning on knowledge graphs created from documents. KGCaRe smartly navigates these graphs to pick out the most helpful facts and checks back for more clues if needed, then uses both graph data and text to generate better answers with explanations. Tests show that KGCaRe works better than several existing methods across different AI models. They also shared their software publicly to help others use this approach.
Large Language ModelsRetrieval-Augmented GenerationKnowledge GraphNeural RetrievalSymbolic ReasoningGraph TraversalVector StoreConditional Question AnsweringPrompt EngineeringDomain-specific QA
Authors
Ghanshyam Verma, Simanta Sarkar, Devishree Pillai, Hotaka Shiokawa, Yourong Xu, Fiona Veazey, Peter Hubbert, Hui Su, Paul Buitelaar
Abstract
Answering complex conditional questions using Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) remains a challenge, particularly in domain-specific contexts where general-purpose LLMs and RAG tend to underperform. We hypothesize that augmenting RAG with unstructured and structured knowledge, extracted from both documents and knowledge graphs (KGs), can improve reasoning and answer accuracy for such tasks. To test this, we propose KGCaRe, a hybrid approach that combines neural retrieval with symbolic reasoning over LLM-generated KGs. KGCaRe constructs a KG from documents using a multi-prompt extraction strategy and stores it in a graph database. Simultaneously, the documents are embedded into a vector store to enable neural retrieval. KGCaRe performs innovative iterative graph traversal guided by the LLM to extract relevant triples, prune irrelevant information, and uses additional clue entities to traverse the graph again if the initial traversal does not provide satisfactory context to generate the answer. The relevant triples extracted from the KG in path form, along with semantically retrieved text passages, are then fed into custom KGCaRe prompts to generate answers to the complex conditional questions with explanations. We evaluate KGCaRe on two complex conditional QA datasets. Our results on these datasets show that KGCaRe consistently outperforms existing baselines, including Vanilla LLM, Code Prompt, Text Prompt, Think-on-Graph, Vanilla RAG, and HybridContextQA, across multiple LLMs such as Mistral, Mixtral, GPT-3.5, and GPT-4o. We publicly release the software pipeline that we developed to implement the proposed KGCaRe approach.