Summary
Many research areas list their unsolved problems only in text documents, which makes it hard to know who is working on what or if problems have been solved recently. The authors built a system that turns these open problems into data by combining a hand-curated list with evidence from scientific papers in one searchable graph. They used an AI model to extract problem details and status from papers but found that automatic claims about problem solutions are often incorrect and must be checked by humans. Their work shows it’s important to keep humans in the loop when updating problem statuses, and they discuss challenges like matching entities and verifying claims.
What this means in practice
- •For library and archive teams: Track and update open research problems across many papers for better knowledge management and discovery.
- •For scientific publishing platforms: Provide researchers and readers with up-to-date structured data about problem status and related research papers in specific fields.$Commercial implications: Enables building advanced research discovery tools that deliver curated problem updates and evidence to users.
Abstract
A research field records its open problems in surveys, keynotes and community reports. Those are prose, written every few years, and nobody can ask them who works on a problem now or whether it was settled last year. We argue the problems should be data, and we build the system that makes them so: a hand-curated problem layer carried in unchanged, under an evidence layer rebuilt from a bibliographic index, joined by typed, provenance-carrying links in one graph database and queryable from one language. A language model does four jobs -- per-paper extraction, catalog grounding, per-paper status verdict, and idea generation -- each constrained to a candidate set so out-of-set answers are refused rather than stored. We report a study on two curated catalogs, database research (1,053 problems) and AI research (1,000 problems), each with a prior catalog-only graph. The rebuilt graph answers every question the catalog-only graph answered -- 47 of 47 on databases, 34 of 35 on AI -- plus questions the catalog-only graph refused: recent workers, untouched problems, methods tried, and outside papers bearing on a problem (336 problems, 770 works on databases). The capability we most wanted does not survive its own audit. Of 41 solves and refutes verdicts hand-checked against the papers, 9 are correct, 30 partly correct and 2 wrong; no refutes verdict (0 of 34) disproves an open problem, and only 1 of 7 solves closes the central question. We report that failure in full and draw the design conclusion: a status change must be proposed with its evidence and never applied, making the human reviewer part of the architecture. We close with the data-engineering problems this raises: entity resolution across curated and harvested layers, verifying model status claims, freshness under source quotas, and evaluation without ground truth.