Crosschapterqa dataset helps ai answer book questions spanning chapters
Reading Between the Chapters: A Graph-Free RAG Rebuilt With AI and Human Feedback
Artificial Intelligence
Summary
Answering questions about a book often requires linking information from chapters far apart, like noticing a sound happening multiple times in a character's life. The authors created CrossChapterQA, a set of questions that need this kind of cross-chapter reasoning across 14 books. They built LadderRAG, a tool that uses summaries and passages instead of graphs to find the right chapters and answer questions. After several redesigns guided by AI and human feedback, the final version retrieves more relevant chapters and produces better answers.
What this means in practice
- •For digital library developers: Create book question answering systems that link information across multiple chapters without complex graph structures.
- •For chatbot designers: Build chatbots that can accurately answer detailed questions about a book by retrieving and summarizing relevant chapters.
Authors
Dipto Sumit, Syeda Fabiayee Jahan, Farig Sadeque
Abstract
Many questions about a book can only be answered by connecting passages chapters apart, such as a cough heard three times across a horse's life. We introduce CrossChapterQA, 1,853 such questions over 14 public-domain books. Each question lists the two to six chapters it needs (spanning half the book at the median) and the relation linking them, cites evidence from each, and has two independent reference answers graded blind. We use it to build LadderRAG, which indexes a book as passages, summaries and book-level cards for about $0.41 and answers a question for under a cent. Its first version failed mostly at retrieval. We redesigned it four times, steered by an LLM judge's verdicts and our analyses of failed answers: a design-level counterpart of RLAIF and RLHF in which feedback changes the architecture, not model weights. We also report the two redesigns that made answers worse. On the 191 questions its first version got wrong, the final version retrieves over twice as many required chapters on held-out books (recall 0.36 to 0.79), and ten annotators rate its answers 6.5/10 on average (764 ratings).