ListQA benchmark tests how well AI lists multiple facts accurately
ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models
Artificial IntelligenceComputation and Language
Summary
Most question-answer tests for AI only check if the model can give one fact, but many real questions need a list of facts. The authors created ListQA, a set of over 9,000 questions that require AI models to provide several pieces of information organized as lists. They tested many language models and found that even the best ones struggle to generate complete and correct lists just from memory but improve a lot when given related Wikipedia pages. This helps show where AI systems need improvement in remembering and organizing detailed factual information.
What this means in practice
- •For ai developers: Evaluate and improve fact retrieval and organization in language models using multi-fact questions and structured list outputs.
- •For knowledge management teams: Use ListQA to test systems that generate structured, multi-element lists for complex fact-based queries across diverse topics and countries.
Authors
Mayank Kulkarni, Rakesh Vaideeswaran
Abstract
Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively target single answer retrieval. We introduce ListQA, a benchmark of 9,045 human-curated, cross-validated questions (audited error rate of 1.7%) that require LLMs to recall multiple facts and compose them into structured lists of 3-10 elements. Questions range from flat lists ("Which NCAA teams went undefeated between 2000 and 2024?") to hierarchical lists with sub-attributes ("...and what was their record and result?"), spanning eight categories across 60+ countries. An evergreen design (explicit temporal constraints or historically immutable facts) keeps ground truth valid without periodic updates. For evaluation, we frame element matching as a linear sum assignment problem: optimal bipartite matching paired with LLM-based semantic grading jointly captures factual accuracy, hallucination, and format compliance. We benchmark 34 models (8 families, 42 configurations across standard and thinking enabled inference) and find: the best standard-inference model scores below 37% LLM-Judge-F1, multi-faceted questions are harder across the board, and providing the source Wikipedia page lifts scores dramatically, confirming that models can organize facts but struggle to retrieve them from parameters alone. Extended analyses cover base vs. instruction-tuned checkpoints and token efficiency. We release the dataset and evaluation code.