Llm based retriever generator improves textual graph question answering
Towards a Learnable Retriever for Textual Graph Question Answering
Computation and LanguageMachine Learning
Summary
Answering questions from graphs made of text is tricky because existing systems either only tweak the language model that generates answers or use unsupervised methods for choosing evidence from the graph. The authors created a system named TaLenT that trains a language model to both find relevant parts in the graph and then generate answers, all in one pipeline. They also teach the system to focus on gold-standard answers and related information while ignoring irrelevant parts, improving the quality of evidence retrieval. Testing on multiple datasets showed that TaLenT does better at finding the right answers.
What this means in practice
- •For nlp system builders: Build systems that answer complex graph-based questions by jointly training retrievers and generators to improve evidence selection and answer accuracy.
- •For knowledge graph platform teams: Enhance question-answering features by integrating learnable retrievers with supervision on gold answers to boost relevant node recall without extra noise.
Authors
Kaifeng Jin, Yuchen Yan, Zhichen Zeng, Xiao Lin, Lihui Liu, Yinglong Xia, Hanghang Tong
Abstract
Textual graph question answering (GraphQA) and graph-based retrieval-augmented generation (GraphRAG) have gained increasing attention as a way to ground large language models (LLMs) on structured knowledge. Despite extensive efforts, existing methods still face two key gaps. The first is an architectural gap: existing methods typically fine-tune an LLM-based generator, but do not also make the retriever itself a learnable LLM for graph evidence selection within one end-to-end retrieve-then-generate pipeline. The second is a retrieval-supervision gap: existing retrievers rarely use gold answer signals to supervise evidence selection. As a result, they may fail to recall evidence nodes required for reasoning, resulting in insufficient coverage of the gold answer set; meanwhile, naive attempts to boost recall often introduce excessive irrelevant information that compromises compactness and overwhelms the LLM. To address these gaps, we propose TaLenT, an LLM-based end-to-end retriever-generator framework for textual GraphQA. To bridge the architectural gap, TaLenT pairs a learnable LLM-based retriever for evidence selection with a fine-tuned LLM-based generator within one unified retrieve-then-generate pipeline. To bridge the retrieval-supervision gap, we leverage gold answer signals as retriever supervision. Specifically, we stratify graph nodes into gold answers, answer-related nodes, and irrelevant noise. We then adopt a weighted pseudo-label loss to prioritize gold answers and preserve answer-related nodes for coverage, while discarding irrelevant noise for compactness. Experimental results across three GraphQA datasets demonstrate that TaLenT consistently improves downstream performance.